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FOREWORD 


The  U.S.  Arny  Research  Institute  Aviation  Research  and 
Development  Activity  (ARIARDA)  at  Fort  Rucker,  Alabama,  is  an 
operational  unit  of  the  U.S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences  (ARI) .  This  work  was  performed  as 
part  of  the  front-end  analysis  work  within  ARIARDA' s  research 
mission.  The  work  was  performed  under  the  Memorandum  of 
Understanding  between  the  U.S.  Army  Aviation  System  command 
(AVSCOM)  and  ARI,  "Establishment  of  Technical  Coordination 
Between  ARI  and  AVSCOM,"  10  April  1985. 

The  potential  impact  of  advanced  technology  on  manpower  and 
personnel  requirements  must  be  considered  when  managing  aviation 
systems.  Since  high  operator  workload  can  result  in  a  dramatic 
decrease  in  system  effectiveness,  workload  is  a  critical 
consideration . 

Over  the  past  8  years,  ARIARDA  has  produced  microcomputer- 
based  models  of  operator  workload  for  existing  and  modified 
versions  of  several  series  of  aircraft.  The  methodology  used  to 
develop  the  models  has  produced  organized  and  useful  results; 
however,  it  has  not  bean  validated  until  now.  This  report 
describes  research  designed  to  test  the  validity  of  the 
methodology. 

The  results  of  the  research  have  been  briefed  to  ARI,  AVSCOM 
personnel,  and  U.S.  Army  Aviation  Center  directorates.  The 
validated  workload  prediction  methodology  should  continue  to  be 
useful  in  evaluating  the  operator  workload  in  Army  systems  well 
into  the  future. 
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PRELIMINARY  VALIDATION  OF  THE  TASK  ANALYSIS /WORKLOAD  METHODOLOGY 


EXECUTIVE  SUMMARY 


R«quirMi«nt: 

The  research  described  in  this  report  %ras  conducted  to 
provide  a  prelininary  indication  of  the  validity  of  the  Task 
Analysis/Workload  (TAWL)  methodology.  Other  objectives  were 

to  detemine  the  differences  in  TAWL  models  produced 
independently  by  two  qualified  analysts  and  to  assess 
the  effect  of  the  differences  on  TAWL  predictions  and 
validity, 

to  evaluate  the  mechanisms  used  in  the  methodology  that 
link  TAWL  predictions  to  operator  performance,  and 

•  to  detemine  the  relationship  between  operator  heart 
rate  variability  and  (a)  objective  perfomance  measures, 
(b)  subjective  ratings  of  workload,  and  (c)  TAWL 
predictions. 


Procedure: 

Because  a  flight  or  simulation  environment  was  unavailable, 
a  set  of  conditions  was  developed  that  produced  a  large  range  of 
operator  workload.  The  equipment  used  to  generate  the  conditions 
included  a  computer  and  display,  a  voice  synthesizer,  a  numeric 
keypad,  a  foot  controller,  and  two  hand  controllers.  A  dichotic 
listening  task,  a  dual-axis  tracking  task,  and  two  independent 
single-axis  tracking  tasks  were  used  singly  and  in  combination  to 
produce  seven  different  conditions  that  varied  in  workload.  Two 
analysts  independently  developed  TAWL  workload  prediction  models 
of  the  seven  conditions.  Twenty  male  AH-64A  aviators  (10  battle- 
rostered  crews)  from  one  battalion  of  an  operational  aviation 
regiment  perfomed  repeated  trials  of  the  task  conditions  over 
three  test  sessions.  Subjective  workload  ratings,  measures  of 
tracking  and  dichotic  listening  perfomance,  and  measures  of 
aviator  heart  rate  variability  were  taken  for  each  trial. 
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Findings : 

The  conduct  of  the  research  resulted  in  the  following 
individual  findings: 

•  Seven  conditions  that  vary  in  workload  can  be 
constructed  from  combinations  of  four  tracking  and 
dichotic  listening  tasks. 

Two  qualified  analysts  were  able  to  use  the  TAHL 
methodology  to  construct  workload  prediction  models  and 
generate  predictions  of  the  workload  in  each  of  the 
conditions. 

Three  types  of  differences  (structural,  rating,  and 
opinion)  were  found  in  the  construction  of  the  two 
models . 

•  The  models'  predictions  differed  but  were  highly 
correlated  (£  •  .99). 

The  subjective  workload  ratings  were  well  organized  and 
sensitive  to  the  task  loading  differences  between  the 
conditions. 

•  The  tracking  performance  consistently  Increased  as  a 
function  of  condition  and  had  reached  a  plateau  by  the 
last  practice  session. 

.  Two  measures  of  dichotic  listening  performance  were  well 
organized  and  sensitive  to  task  condition. 

•  The  heart  rate  variability  measures  were  not  well 
organized  or  sensitive  to  task  condition. 

•  Three  measures  showed  sufficient  sensitivity  to  the 
across-condition  workload  manipulation  to  be  used  as 
criteria  for  TAWL  validity. 

•  The  correlations  among  the  three  criterion  measures  and 
the  average  of  TAWL  component  predictions  were  high 
(.89  <  £  <  .99)  for  both  models. 

These  findings  led  to  the  following  conclusions: 

.  The  TAWL  methodology  has  excellent  potential  for 
generating  valid  predictions  of  operator  workload. 

•  Although  differences  were  observed  between  the  models 
generated  by  different  analysts,  the  differences  did  not 
reduce  the  validity  of  either  model. 
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•  Although  the  relationship  between  operator  workload  and 
performance  is  complex  (even  small  amounts  of  workload 
can  sometimes  degrade  performance) ,  the  TAWL  methodology 
provides  an  excellent  description  of  the  aspects  of  the 
task  environment  known  to  effect  operator  performance. 

«  The  validation  of  the  methodology  should  continue  with 
the  validation  of  a  TAWL  model  of  full  complexity. 


Utilization  of  Findings: 

The  findings  support  the  validity  of  the  TAWL  methodology 
for  producing  valid  models  of  operator  workload.  Operator 
workload  should  continue  to  be  a  critical  consideration  in  Army 
system  design  well  into  the  future. 
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PRELIMINARY  VALIDATION  OP  THE 
TASK  ANALYSIS/WORKLOAO  METHODOLOGY 

Introduction 

A  methodology  was  developed  for  predicting  operator 
workload  using  the  information  from  a  task  analysis  of  the 
system.  The  original  methodology  was  developed  for  use  in 
the  concept  exploration  and  definition  phase  of  the  Army's 
light  helicopter/  experimental  <LHX)  development  program 
(Aldrich,  Craddock,  &  McCracken,  1984;  Aldrich,  Srabo,  & 
Craddock,  1986;  McCracken  £  Aldrich,  1984) .  Analyses  were 
conducted  to  compare  the  operator  workload  of  one-  and  two- 
crewmember  configurations  of  the  LHX. 

Subsequently,  a  comprehensive  tcsk/workload  analysis  of 
all  phases  was  conducted  for  the  AH-64A  attack  mission.  The 
results  of  the  analysis  are  described  in  a  technical  report 
by  Szabo  and  Bierbaum  (1986)  .  ^plying  the  methodology  to  an 
operational  aircraft  (AH-64A)  allowed  a  far  greater  degree  of 
specificity  in  the  task  element  descriptions  than  was 
possible  during  the  concept  exploration  and  definition  phase 
of  the  LHX  program. 

The  methodology  used  to  perform  the  AH-64A  analysis  was 
further  refined  and  used  to  predict  the  effect  on  operator 
workload  of  modifications  to  Army  special  operations 
helicopters.  The  methodology  was  used  to  predict  the 
crewmember  workload  for  existing  and  modified  versions  of  the 
UH-60A  (Bierbaum  &  Hamilton,  1990;  Bierbaum,  Szed?o,  £ 

Aldrich,  1989) ,  and  CH-47D  aircraft  (Bierbaum  &  Aldrich, 

1989;  Bierbaum  &  Hamilton,  1991) . 

The  refined  version  of  the  methodology  is  called  the 
task  analysis/workload  (TAWL)  methodology.  Computer  support 
for  the  methodology  has  been  developed  and  named  the  TAWL 
operator  simulation  system  (TOSS) .  Hamilton,  Bierbaum,  and 
Fulford  (1991)  provide  a  detailed  description  of  both  the 
TAWL  methodology  and  the  TOSS  software. 


The  TAWL  Methodology  Overview 

A  TAWL  workload  prediction  model  is  developed  in  three 
stages.  In  the  first  stage,  the  analyst  performs  a  task/ 
workload  analysis  on  the  system.  A  prototype  mission  for  the 
system  is  developed  and  is  progressively  decomposed  into 
phases,  segments,  functions,  and  tasks.  The  analysis  yields 
estimates  of  the  duration  of  tasks,  a  description  of  the 
sequence  of  tasks,  and  a  description  of  the  crewmember  and 
subsystem  associated  with  each  task.  The  workload  analysis. 
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based  on  a  multiple  resources  theory  of  human  attention, 
yields  independent  estimates  of  the  cognitive,  psychomotor, 
and  sensory  components  of  workload  (hereafter  referred  to  as 
workload  components)  for  each  task.  The  theory  differs  from 
other  multiple  resource  theories  of  attention  in  the  nature 
and  number  of  components  that  are  identified.  The  theory 
underlying  the  TAWL  methodology  recognizes  five  independent 
workload  components:  auditory,  kinesthetic,  visual, 
cognitive,  and  psychomotor.  See  Wickens  (1984)  for  a  review 
of  multiple  resource  theories  of  attention  and  their  relation 
to  workload. 

The  TAWL  methodology  treats  each  of  the  workload 
components  independently  for  two  reasons.  First,  although 
interactions  among  the  components  probably  occur,  the  nature 
of  the  Interactions  cannot  be  defined  adequately  at  this 
time.  Second,  the  information  that  results  from  treating 
workload  components  individually  is  useful  for  identifying 
potentially  effective  ways  to  reduce  workload  or  to 
redistribute  workload  among  the  crewmembers,  subsystems,  or 
components.  For  example,  a  designer  ^an  decide  whether 
additional  information  should  be  presented  visually  or 
aurally  by  determining  which  component  has  the  least  amount 
of  workload. 

The  workload  analysis  is  based  on  subjective  estimates 
of  operator  workload  rather  than  on  estimates  derived  through 
experimentation.  Research  analysts  and  subject  matter 
experts  (SMEs)  generate  workload  estimates  by  using  equal- 
interval,  verbally  anchored  rating  scales;  the  scale  values 
range  from  0  to  7 .  This  approach  avoids  the  large 
expenditures  of  time,  money,  and  manpower  that  would  be 
required  to  derive  empirical  measures  of  workload  for  each 
task . 


In  the  second  stage  of  the  TAWL  methodology,  the  analyst 
develops  a  model  of  each  crewmember's  actions  by  recombining 
tasks  to  simulate  the  behavior  of  the  crewmembers  during  each 
segment  of  the  mission.  Function  decision  rules  are 
developed  that  describe  the  sequencing  of  tasks  within  each 
function;  segment  decision  rules  are  developed  that  describe 
the  start  time,  stop  time,  and  interaction  of  the  functions 
within  each  segment.  The  underlying  assumption  is  that  the 
segments  can  be  combined  to  model  the  crewmembers'  behavior 
for  individual  mission  phases  and  for  the  entire  mission. 

In  the  third  stage  of  the  TAWL  methodology,  the  analyst 
executes  the  model  to  simulate  the  crewmembers'  actions 
during  the  operation  of  the  system.  The  TOSS  computer 
software  performs  the  simulation  and  produces  estimates  of 
each  crewmember's  cognitive,  psychomotor,  and  sensory 
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workload  for  each  half-second  of  the  mission.  The  estimates 
of  workload  for  each  component  are  generated  by  summing  the 
workload  for  that  component  across  all  tasks  that  the 
cretmember  performs  during  each  half-second  of  the  mission. 

For  example,  during  a  specific  half-second  interval,  the 
pilot  performs  the  following  tasks:  Control  Attitude,  Check 
External  Scene,  and  Transmit  Communication.  The  cognitive 
workload  for  the  three  tasks  during  that  interval  is  1.0, 

1.0,  and  5.3,  respectively.  Thus,  the  estimate  of  cognitive 
workload  for  the  pilot  during  that  interval  is  7.3.  A 
predefined  overload  threshold  is  used  during  execution  of  the 
model  to  measure  the  amount  of  time  during  the  mission  that 
each  crewmember  experiences  an  overload  condition. 

Using  the  TAWL  prediction  methodology,  an  analyst  can 
develop  a  model  of  a  system  and  use  the  model's  output  to 
determine: 

•  the  absolute  and  relative  workload  of  the  crewmember, 

•  the  time  intervals  during  which  crewmembers  experience 
high  workload,  and 

•  the  components  for  which  crewmembers  experience  high 
workload. 

The  TAWL  methodology  yields  sufficient  information  to 
enable  system  designers  tc  reduce  or  redistribute  workload 
over  time,  crewmembers,  or  components.  Designers  also  may 
use  the  information  to  identify  design  alternatives  that 
result  in  lower  workload.  In  addition  to  the  uses  described 
above,  the  methodology  yields  mission  timelines  and  task 
listings  that  can  be  used  to  develop  the  system's  manning  and 
training  requirements. 


TAWL  Validity  Research 

The  only  research  conducted  to  test  the  validity  of  the 
predictions  generated  by  a  TAWL  workload  prediction  model  was 
performed  by  lavecchia,  Linton,  Bittner,  and  Byers  (1989) . 

The  researchers  collected  subjective  ratings  of  overall 
workload  (OW)  and  peak  workload  (PW)  from  seven  UH-60A  crews 
for  six  segments  of  a  mission  flown  in  the  UH-60A  flight 
simulator.  They  used  the  UH-60A  workload  prediction  model 
(Bierbaum  &  Hamilton,  1990;  Blerbaum,  Szabo,  &  Aldrich,  1989) 
to  compute  OW-comparable  TAWL  predictions  by  averaging  across 
the  six  workload  components  (i.e.,  cognitive,  psychomotor, 
auditory,  visual-unaided,  visual-aided,  and  kinesthetic)  and 
each  half-second  of  segment  time.  They  prepared  PW- 
comparable  TAWL  predictions  by  summing  across  workload 
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components  for  each  half-second  of  segment  time  and  by 
selecting  the  iTiaxlmum  sxim  for  each  mission  segment. 


The  TAWL  predictions  correlated  highly  with  OW  for  the 
12  unique  conditions  (x  *.82) .  The  researchers  noted  that 
the  TAWL  predictions  were  consistent  with  all  of  the  copilot 
OW  ratings  and  all  but  one  of  the  pilot  ratings.  The  pilot 
communication  in  this  segment  was  not  as  con^lex  as  assumed 
in  the  model.  Subsequently,  this  segment  was  dropped  from 
the  analysis  and  the  correlation  was  recomputed  using  the 
remaining  11  segments;  the  correlation  coefficient  increased 
to  .95.  The  correlations  between  TAWL  and  PW  were  lower,  x  » 
.62.  Based  upon  these  results,  lavecchia  et  al.  (1989) 
concluded,  "The  TAWL  model  has  substantial  potential  as  an 
analytical  wor)cload  estimation  technique  which  may  be  applied 
before  system  development." 


Research  Obieetive 

Although  the  results  of  lavecchia' s  research  (lavecchia 
et  al.,  1989)  support  the  validity  of  TAWL  predictions,  the 
research  has  four  important  limitations.  First,  the  research 
employed  only  subjective  worlcload  ratings  as  validation 
criteria.  Second,  the  subjective  worlcload  ratings  used  as 
validation  criteria  (OW  and  PW)  are  based  on  each  aviator's 
own  Internal  (unknown)  reference  and  not  on  a  standard 
reference  task.  Third,  the  relationship  among  TAWL 
predictions  and  subjective  workload  ratings  was  determined 
for  a  limited  range  of  task  loadings.  Finally,  a  relatively 
small  number  of  different  conditions  was  examined  in  the 
research.  The  present  research  addresses  the  first  three  of 
the  limitations  listed  above,  but  not  the  fourth. 
Specifically,  the  objective  of  the  present  research  was  to 
assess  the  validity  of  TAWL  predictions  for  a  range  of  task 
loading  using  both  objective  performance  measures  and  a  well 
established  subjective  rating  of  workload — the  NASA  Task  Load 
Index  (TLX) — as  validation  criteria. 

The  present  research  had  three  additional  objectives. 
First,  determine  the  differences  in  TAWL  models  produced 
independently  by  two  qualified  analysts  and  assess  the  effect 
of  the  differences  on  TAWL  predictions  and  validity.  Second, 
evaluate  the  mechanisms  used  in  the  methodology  that  link 
TAWL  predictions  to  operator  performance  based  on  the  results 
of  the  research  and  refine  the  methodology.  Third,  determine 
the  relationship  between  operator  heart  rate  variability  and 
(a)  objective  performance  measures,  (b)  subjective  ratings  of 
workload  (TLX),  and  (c)  TAWL  predictions.  Because  heart  rate 
variability  has  been  proposed  as  a  valid  and  sensitive 
measure  of  workload  (Aasman,  Mulder,  &  Mulder,  1987),  the 
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present  research  was  used  as  an  opportunity  to  explore  the 
ease  of  use  and  sensitivity  of  the  measure  In  the  aviation 
task  environment.  If  heart  rate  variability  proved  to  be  a 
sensitive  and  viable  measure  of  the  task  load  manipulations 
of  interest,  it  could  be  used  in  future  research  to  provide  a 
continuous,  non-intrusive  measure  of  workload  that  could  be 
acquired  in  flight. 


Method 

This  section  describes  the  methods  used  to  assess  the 
validity  of  the  TAWL  workload  prediction  methodology.  The 
following  eight  subsections  describe  the  experimental  tasks, 
model  development,  TAWL  predictions,  materials,  aviators, 
apparatus,  procedures,  and  data  analysis. 


Experimental  Tasks 

Task  aeiecrion.  To  examine  the  relationship  among  the 
variable.*}  of  interest  over  a  wide  range  of  conditions,  a  set 
of  tasks  was  needed  that  produces  a  large  range  of  operator 
workload.  Fortunately,  the  U.  S.  Army  Aviation  Center 
(USAAVMC)  uses  just  such  a  set  of  tasks  to  aid  in  the 
assignment  of  aviators  to  aircraft  after  they  have  completed 
initial  entry  rotary  wing  (lERW)  training  (Intano,  Howse,  & 
Lofaro,  1991a;  Intano,  Howse,  &  Lofaro,  1991b;  Intano  & 
Lofaro,  1989;  Intano  &  Lofaro,  1990) .  The  tasks  were 
developed  from  research  conducted  by  the  Israeli  Air  Force 
(Gopher  &  Kahneman;  1971)  and  were  adapted  by  the  Navy  for 
use  in  aviator  selection  (Griffin  &  MacBrlde;  1986) .  The 
tasks  are  controlled  by  a  computer  program  written  in  the 
Basic  computer  language.  The  equipment  used  to  generate  the 
tasks  included  a  computer,  a  computer  display,  a  voice 
synthesizer,  a  numeric  keypad,  a  foot  controller,  and  two 
hand  controllers.  Different  conditions  are  produced  by  using 
four  basic  tasks  singly  and  in  combination.  The  four  tasks 
are  a  dichotic  listening  task,  a  dual**axis  tracking  task,  and 
two  independent  single-axis  tracking  tasks. 

Dichotie  listening  tasks.  Seven  conditions  were 
constructed  from  combinations  of  the  four  basic  tasks.  The 
four  basic  tasks  and  the  seven  conditions  are  described 
below. 

In  the  dichotic  listening  (DL)  task,  standard  audio 
headphones  were  used  to  present  simultaneously  different 
strings  of  letters  and  digits  to  each  ear.  The  aviators  were 
instructed  to  attend  only  to  the  information  presented  to  one 
ear  and  to  enter  on  a  numeric  keypad  the  digits  in  that 
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string.  Each  trial  lasted  3  minutes.  During  each  trial,  8 
DL  presentations  were  made  using  16  22-character  strings  for 
each  ear.  Each  DL  presentation  consisted  of  the  27  events 
listed  in  Table  1.  A  trial  consisted  of  an  initial  period  of 
silence,  8  stimulus  presentations,  and  another  period  of 
silence  at  the  end. 

To  remain  consistent  with  the  other  DL  tasks  being  used 
by  the  USAAVNC,  the  stimulus  strings  were  presented  and 
analyzed  in  two  parts.  The  first  part  (16  characters) 
consisted  of  randomly  dispersed  digits  and  letters;  the 
second  part  (6  characters)  always  consisted  of  2  letters 
followed  by  4  digits.  This  pattern  was  obvious  to  the 
participants  and  was  used  in  their  tasJc  performance 
strategies . 

For  each  DL  trial,  a  random  number  from  1  to  36  was 
selected  that  defined  the  starting  point  in  the  36  record  DL 
stimulus  file.  The  36  DL  stimulus  strings  used  in  the 
experiment  are  presented  in  Table  2 . 

Tracking  taslcs .  The  remaining  three  basic  tasks  were 
continuous,  compensatory  tracking  tasks  that  required  the 
aviators  to  use  a  control  to  keep  a  moving  cursor  aligned 
with  a  stationary  target.  The  three  tracking  tasks  differed 
in  the  number  of  dimensions  in  which  the  cursor  could  move 
(one  or  two  dimensions) ,  the  direction  of  the  cursor  movement 
(horizontal,  vertical,  or  both),  and  the  design  of  the 
control  (single  axis  joystick,  dual  axis  joystick,  or 
pedals) .  The  target  and  cursor  for  all  three  tracking  tasks 
were  presented  on  the  same  color  computer  display.  The 
stimulus  configuration  for  the  three  tracking  tasks  is 
illustrated  in  Figure  1  and  described  below. 

The  target  and  cursor  for  the  dual-axis  tracking  task 
appeared  within  a  large  rectangle  located  in  the  upper  left 
portion  of  the  display.  The  target  consisted  of  a  large 
cross  created  by  a  vertical  dashed  line  and  a  horizontal 
dashed  line  that  intersect  at  the  center  of  the  rectangle; 
the  cursor  was  a  small  cross  created  with  solid  lines  (see 
Figure  1) .  The  cursor  was  controlled  with  a  dual-axis 
joystick.  The  joystick  was  mounted  between  the  aviator's 
legs  in  a  manner  similar  to  the  control  stick  of  a  fixed-wing 
aircraft.  The  pivot  point  of  the  dual-axis  joystick  was 
located  only  a  few  inches  below  the  thigh  level.  The  dual- 
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Table  1 


Timing  Sequence  of  a  Single  Dlchotlc  Listening  Presentation 


Sequence 

Time 

(sec) 

Event 

1 

.70 

"TEST" 

2 

1.39 

"LEFT"  or  "RIGHT" 
(Instruction  for  which  ear 

to 

attend) 

3 

2.09 

1st  character  In  part  1 
(stop  collecting  responses 

for 

part 

2) 

4 

2.79 

2nd  character  In  part  1 

5 

3.48 

3rd  character  In  part  1 

6 

4.18 

4th  character  in  part  1 

7 

4.88 

5th  character  In  part  1 

8 

5.58 

6th  character  In  part  ] 

9 

6.27 

7th  character  In  part  1 

10 

6.97 

8th  character  in  part  1 

11 

7.67 

9th  character  In  part  1 

12 

8.36 

10th  character  In  part  1 

13 

9.06 

11th  character  in  part  1 

14 

9.76 

12th  character  in  part  1 

15 

10.45 

13th  character  In  part  1 

16 

11.15 

14th  character  In  part  1 

17 

11.85 

15th  character  In  part  1 

18 

12.55 

16th  character  In  part  1 

19 

13.24 

silence 

20 

13.94 

silence 

21 

14.64 

1st  character  In  part  2 
(stop  collecting  responses 

for 

part 

1) 

22 

15.33 

2nd  character  In  part  2 

23 

16.03 

3rd  character  In  part  2 

24 

16.73 

4th  character  in  part  2 

25 

17.42 

5th  character  In  part  2 

26 

18.12 

6th  character  In  part  2 

27 

20.52 

silence 

Note.  Each  event  from  sequence  1  to  26  used  23  ticks  of  the  conputer 
clock  («.7  s)  except  for  action  number  27  which  used  79  ticks  (*2.4  s) . 
Time  is  approximate  elapsed  time  in  seconds. 
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Table  2 


The  36  Stimulus  Strings  Used  In  the  Dlchotlc  Listening  Task 


Stimulus 

Left  Ear 

Right  Ear 

Part  1 

Part  2 

Part  1 

part  2 

1 

GZy3FMF49GOSL01F 

BZ2741 

B8FXOF2LS7GR6Xy5 

FY9630 

2 

L09NYBN17PXFS3SR 

GX6381 

M8ZGXP0PO2S66MF4 

LS7290 

3 

2XN9FF0HY6SL3OBG 

MR9438 

S48RBXY75NF6LSF1 

N06152 

4 

M43LFXN6Sy7602LF 

ZB2897 

9ZY86F1OR0SF5RNX 

YF6413 

S 

Z6yO3FLZ0)CM9NL7F 

XG2147 

2LXNMyF4S51RGBR8 

SL0936 

6 

yF7FLRG130N0R946 

RM4126 

X2NSyM8LF6BX0Fy5 

ON5398 

7 

GFSFlKy85RN0FR9Z 

BZ0927 

LY7GX63LP2SMOXN4 

FY1836 

8 

L4SG0FZX6NO2XFM8 

6X2516 

5R0FR1 9NS37LyBLO 

LS8349 

9 

FlXN076RLy5MFFS4 

HR3146 

BGSL0ML93Oy6XN2Z 

H079B2 

10 

F5FOeLMS0FG6yMX2 

RB6390 

4GKF017NL930GRYF 

SF7421 

11 

L81R04XNGBSSMX2L 

LH893S 

7 YF 9LNyOMLeF3RZ 6 

X06214 

12 

0SLR20FX60S7GRB8 

6R0369 

lOMSF43yB09GFXFZ 

FN1472 

13 

lFL0OS9GF4FMy3GZ 

OG4673 

y56XGRS72LOFFXB8 

SL2109 

14 

SRS3XF7FNiyB9NLO 

Nr9753 

R46MSGO20FXFZ6M8 

OG6418 

15 

BG3OSLy60HFfN92X 

yM4135 

FlLXF6SNy7BS8RS4 

NB6287 

16 

LF027GSyN6FX3LM4 

SN1862 

NX5RSFR01OGFY89Z 

RO4095 

17 

7FNLM90XLZ3FyOZ6 

XL4609 

R8GB1RS5F4MYXH2L 

FR5827 

18 

4GR9K030GlLR7FyF 

XL1937 

y50FBSF68LYMNSX2 

BM2845 

19 

9ZFRN05Ry81NSFGF 

Ry4681 

N40XSMF23LX67GLX 

LS7935 

20 

M8XFO26NZX0FSGL4 

NZ2768 

L0YB7LS39NR10F5R 

R01453 

21 

S 4 FF 5MLy 8 R07XNF 1 

RH04S9 

2ZXNY63OL90MSLBG 

L08126 

22 

X2yMG60FMS8LFOF5 

GS728S 

YFGR30L97N01NF4G 

XY9064 

23 

2LMXSSBGXN041RL8 

SB6485 

Z 6 3R0 FMLYOLNF 97 Y 

MR7391 

24 

B8GRS7  60FX20LR0S 

XF2190 

FZFX96B03yF4MSlO 

YB6473 

25 

B8FXOF2LS7GR6Xy5 

FY9630 

GZY3FMF49GO5L01F 

BZ2741 

26 

M8  ZGXF0FO2  SG6MF  4 

LS7290 

LO 9NYBN1 7 FXFS 3 SR 

GX6361 

27 

S48RBFy75NFGLOFl 

N06152 

2XN9FS0MY6SL3ZBG 

MR9438 

28 

9ZY8GF1OR0SF5RNF 

yF6413 

M43LFXN6SY7G02LS 

ZB2897 

29 

2LXNMyF4S51RGBR8 

SL0936 

Z6YO3FLZ0XM9NL7F 

XG2147 

30 

X2NSyM8LF6BO0Fy5 

ON5398 

YF7FLRG130NXR94G 

RM4126 

31 

LF7GX63LF2SMOXN4 

Fyi836 

GFSF1NY8SRN0FR9Z 

BZ0927 

32 

SROFR19NS37LyBLO 

LS6349 

L4SG0FZX6NO2XFM8 

GX2516 

33 

BGSL0ML93oy6XN2Z 

N07982 

F1XN078RLY5MFFS4 

MR3146 

34 

4GNF017NL930GRyF 

SF7421 

F5FO8LMS0FG6yMX2 

RB6390 

3b 

7yF 9LNyOML0F3RZ 6 

X06214 

L81R04XNGBS5MX2L 

U48935 

36 

1OMSF43YB09GFXFZ 

FN1472 

05LR20FX60S7GRB8 

GR0369 

8 


Computer  Disf^ay 


Figure  1.  Tracking  task  display  configuration.  (The 
annotation,  added  here  for  descriptive  purposes,  was  not 
present  on  display) . 


axis  joystick  moved  in  the  same  planes  as  an  aircraft  control 
stick  (right-left  and  fore-aft) .  Movement  of  the  joystick  in 
the  right-hand  (left-hand)  direction  caused  the  cursor  to 
move  left  (right);  movement  of  the  joystick  in  the  forward 
(backward)  direction  caused  the  cursor  to  move  down  (up) . 

The  target  and  cursor  for  the  horizontal-axis  tracking 
task  appeared  in  a  rectangle  located  on  the  lower  portion  of 
the  display,  directly  beneath  the  large  rectangle  in  which 
the  dual-axis  target  and  cursor  were  displayed.  As  shown  in 
Figure  1,  both  the  target  and  cursor  appeared  as  solid 
vertical  bars  of  equal  height;  however,  the  cursor  bar  was 
about  twice  as  wide  as  the  target  bar.  The  cursor  for  the 
horizontal-axis  tracking  task  was  controlled  by  a  set  of 
pedals  similar  in  location  to  the  pedals  used  in  an  aircraft 
to  control  yaw.  Pressing  the  left-hand  pedal  caused  the 
cursor  to  move  to  the  right;  pressing  the  right-hand  pedal 
caused  the  cursor  to  move  left . 

The  target  and  cursor  for  the  vertical-axis  tracking 
task  appeared  in  a  rectangle  located  on  the  left  portion  of 
the  display.  The  target  and  cursor  used  for  the  vertical- 
axis  tracking  task  were  the  same  size  and  shape  as  the  target 
and  cursor  used  for  the  horizontal-axis  tracking  task,  but 
the  bars  were  oriented  horizontally  (see  Figure  1) .  The 
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cursor  was  controlled  with  a  single-axis  joystick  located  to 
the  left  of  the  aviator  in  about  the  same  position  as  a 
fixed-wing  aircraft  throttle.  The  single-axis  joystick  was 
identical  to  the  dual-axis  joystick  and  moved  in  a  fore-aft 
and  right-left  directions;  However,  only  fore-aft  movement 
controlled  cursor  movement.  The  joystick's  pivot  point  was 
located  only  a  few  inches  below  the  aviator's  hand. 

Movement  of  the  joystick  in  the  forward  direction  caused  the 
cursor  to  move  down;  an  aft  movement  of  the  joystick  caused 
the  cursor  to  move  up.  Aviators  were  instructed  to  use  the 
single-axis  joystick  to  keep  the  wide  bar  (cursor)  aligned 
with  the  narrow  bar  (target) . 

To  improve  the  statistical  power  of  tracking  performance 
data  for  comparisons  of  the  different  conditions  used  in  the 
research,  the  characteristics  of  the  dual-axis,  horizontal, 
and  vertical  forcing  functions  were  carefully  controlled. 

For  each  tracking  task  the  forcing  function  (the  computer 
controlled  movement  of  the  cursor)  was  determined  by  the  data 
stored  in  a  parameter  file.  The  contents  of  the  three  files 
have  been  combined  and  translated  in  Table  3.  The  files  were 
generated  in  two  stages.  In  the  first  stage,  a  pseudo-random 
series  of  integers  from  3  to  10  were  generated  for  each 
tracking  task  (dual-axis,  horizontal,  and  vertical) .  Each 
series  summed  to  the  180  and  was  used  to  define  the  number  of 
seconds  in  discrete  time  periods  during  the  180  second  trial 
in  which  the  cursor  would  be  forced  in  a  particular 
direction.  The  first,  third,  and  fifth  columns  of  Table  3 
contain  the  discrete  time  periods  of  the  forcing  function 
used  in  the  experiment. 

In  the  second  stage,  another  pseudo-random  series  of 
-Is,  Os,  and  +la  was  generated  for  each  tracking  dimension 
for  each  tracking  task.  These  series  of  numbers  were  used  to 
define  the  direction  of  cursor  movement  in  the  following 
manner.  For  horizontal  tracking  tasks  -1  indicated  a 
leftward  movement,  +1  indicated  a  rightward  movement,  and  0 
indicated  no  horizontal  movement.  For  vertical  tracking 
tasks  -1  indicated  a  downward  movement,  +1  indicated  an 
upward  movement,  and  0  indicated  no  vertical  movement.  The 
second,  fourth,  and  sixth  columns  in  Table  3  contain  the 
direction  of  cursor  movement  used  for  each  tracking  task. 

The  forcing  function  for  each  tracking  task  was  exactly 
the  same  for  each  experimental  trial.  For  each  display  frame 
generated,  the  computer  program  moved  the  cursor  one  pixel  in 
the  direction  specified  by  Table  3.  Thus,  for  the  first 
three  seconds  of  each  trial  in  which  dual-axis  tracking  was 
being  performed,  the  cursor  moved  left  one  pixel  and  down  one 
pixel  33  times  a  second;  for  the  next  7  seconds,  the  cursor 


10 


Table  3 


Forcing  Functions  for  the  Dual-Axis,  Horizontal,  and  Vertical 
Tracking  Tasks 


Dual-Axis  Horizontal  Vertical 


Seconds  Direction  Seconds  Direction  Seconds  Direction 


3 

Left-Down 

6 

None 

€ 

Up 

7 

None-None 

6 

Right 

6 

Up 

7 

Right -Do%m 

4 

Left 

4 

Up 

5 

Hone-Up 

3 

Left 

3 

None 

4 

Left -None 

4 

Left 

5 

up 

4 

Hone -Up 

3 

Left 

8 

Up 

5 

Right -Do»m 

4 

None 

5 

Up 

4 

None -None 

4 

Left 

3 

None 

€ 

Right -Up 

5 

Left 

5 

Down 

3 

Right -None 

7 

None 

5 

Down 

4 

None -Up 

6 

None 

4 

None 

7 

Left-None 

7 

Right 

8 

Up 

6 

None-Down 

8 

Left 

5 

None 

4 

Right -Do»m 

6 

None 

8 

Down 

5 

Right -None 

3 

Right 

6 

None 

3 

Left-None 

5 

Right 

5 

Down 

8 

None-up 

3 

Right 

7 

None 

€ 

Left-Down 

6 

Right 

8 

Up 

6 

Right-Down 

5 

None 

7 

None 

8 

None-Dotm 

7 

Right 

6 

Down 

7 

Right-Down 

8 

Right 

6 

None 

8 

Left-None 

7 

Right 

6 

Up 

8 

None-Down 

4 

None 

3 

None 

5 

Right-Down 

4 

Left 

5 

Up 

6 

None -None 

5 

Left 

€ 

Up 

4 

Right -None 

4 

None 

4 

None 

3 

None-up 

6 

Right 

6 

Down 

8 

Left -Down 

6 

None 

7 

None 

5 

Left-None 

3 

Left 

8 

Up 

5 

None-up 

5 

None 

8 

Up 

6 

None-up 

7 

Left 

7 

Down 

10 

Right -Up 

7 

None 

10 

Right 

was  not 

forced  in  any 

direction . 

Although  the 

aviators 

performed  many  repeated  trials,  the  consistency  of  the 
forcing  function  was  not  apparent  to  the  participants. 
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Ta5^1c  conditions.  Seven  conditions  were  constructed 
using  the  four  basic  tasks.  Rather  than  simply  assigning 
each  task  an  arbitrary  number,  each  condition  is  identified 
with  a  four  character  string;  one  character  signifies  the 
presence  or  absence  of  each  of  the  four  basic  tasks.  If  the 
DL  task  was  performed  in  the  condition,  the  letter  D  is  used 
in  the  first  position.  If  the  dual-axis  tracking  task  was 
performed,  the  letter  S  (taken  from  the  word  stick)  is  used 
in  the  second  position.  Similarly,  if  the  horizontal-  and 
vertical-axis  tracking  tasks  were  performed,  the  letter  P 
(taken  from  the  word  pedal)  and  the  letter  T  (taken  from  the 
word  throttle)  are  used  in  the  third  and  fourth  positions, 
respectively.  The  letter  x  replaces  its  characteristic 
letter  when  a  task  is  not  performed  in  the  condition. 

Using  the  coding  scheme  described  above,  the  seven 
conditions  created  for  this  research  are  xSxx,  Dxxx,  DSxx, 
xSPx,  DSPx,  xSPT,  and  DSPT.  The  seven  task  combinations  were 
selected  from  24  possible  combinations  of  the  four  tasks  to 
provide  a  large  variation  in  workload.  Furthermore,  if  the 
tracking  tasks  are  considered  collectively  as  three  levels  of 
a  single  psychomotor  variable,  the  selected  conditions  cover 
all  possible  combinations  of  a  two  level  DL  variable  and  a 
four  level  psychomotor  variable  (with  the  notable  exclusion 
of  the  xxxx  combination) . 

The  DSPT  condition  requires  the  aviators  to  perform 
three  tasks  with  two  hands.  The  dual-axis  tracking  task, 
vertical  tracking  task,  and  the  DL  response  entry  required 
the  use  of  the  operator's  hands.  This  condition  was  included 
with  the  expectation  that  the  situation  would  lead  to 
operator  overload  and  therefore  provide  valuable  information 
on  pilot  strategies  and  performance  during  overload 
conditions . 


Model  Development 

To  test  the  inter-analyst  reliability  and  the  robustness 
of  the  TAWL  methodology,  two  analysts  independently  developed 
TAVn,  workload  prediction  models  of  the  seven  conditions.  One 
analyst  had  2  years  experience  in  workload  modeling  and  a  PhD 
level  education  in  experimental  psychology;  the  other  was  a 
retired  helicopter  pilot  who  had  7  years  experience  in 
workload  modeling  and  a  masters  level  education.  Each  of  the 
models  were  constructed  with  the  TOSS  software  in  accordance 
with  the  instructions  given  in  Hamilton,  Bierbaum,  and 
Fulford  (1991) .  The  models  are  referred  to  hereafter  as  TAWL 
Model  1  and  TAWL  Model  2,  respectively. 
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TAWL  Predictions 


At  the  end  of  the  simulation  of  each  condition,  TOSS 
computed  the  peak,  mean,  and  standard  deviation  for  the  half- 
second  workload  predictions.  In  addition,  TOSS  identified 
the  intervals  during  which  the  performance  of  concurrent 
tasks  resulted  in  excessive  workload  (referred  to  hereafter 
as  overload) .  The  three  indexes  of  overload  computed  by  TOSS 
and  used  in  this  research  are  described  in  the  following 
paragraphs . 

Component  overload.  A  component  overload  occurs  when 
the  total  workload  for  a  single  component  reaches  or  exceeds 
a  value  of  8  during  a  half-second  Interval  of  the  task 
simulation.  Thus,  several  component  overloads  (i.e., 
cognitive,  psychomotor,  visual,  etc.)  could  occur  for  each 
half-second  Interval.  Because  the  maximum  value  oi.  the  7- 
point  workload  component  rating  scales  requires  all  the 
aviator's  attention  for  that  component,  the  value  8  was 
chosen  as  the  overload  threshold. 

Overload  condition.  An  overload  condition  exists  during 
the  period  of  time  when  at  least  one  component  overload 
occurs.  A  new  overload  condition  is  counted  when  the  tasks 
contributing  to  a  component  overload  change.  Overload 
conditions  identify  the  unique  task  environments  that 
generate  one  or  more  component  overloads. 

Overload  dgnaity.  Overload  density  is  the  percentage  of 
time  during  a  condition  that  a  component  overload  is  present. 
Overload  density  is  computed  by  dividing  (a)  the  number  of 
half-second  intervals  that  contain  component  overloads  by  (b) 
the  total  number  of  half-second  intervals  in  the  condition. 


Materials 

Demographic  questionnaire.  The  aviator  demographic 
questionnaire  (see  Appendix  A)  was  designed  to  collect 
personal,  training,  and  flight  experience  data  for  the 
aviators  who  participated  in  the  research.  All  aviators 
completed  the  questionnaire  during  the  initial  briefing. 

NASA-Task  Load  Index  (TLX) .  The  NASA-Task  Load  Index 
(TLX)  is  a  method  that  NASA  developed  to  measure  the 
subjective  workload  experience  of  an  operator.  NASA-TLX 
measures  are  generated  by  requiring  the  system  operator  to 
rate  the  mental  demand  (MD) ,  physical  demand  (PD) ,  temporal 
demand  (TD) ,  performance  (PF) ,  effort  (EF) ,  and  frustration 
(FR)  Imposed  by  the  system.  The  ratings,  collected  on 
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20-step  bipolar  scales,  yield  scores  from  0  to  100  assigned 
to  the  nearest  5. 

The  subscale  ratings  are  weighted  for  each  individual 
and  are  combined  into  a  single  rating  of  workload.  The 
individual  weights  are  determined  from  a  paired  comparison 
task  that  is  performed  before  the  workload  assessments  are 
made.  Paired  comparisons  require  the  operator  to  choose 
which  subscale  is  more  relevant  to  workload  for  each  of  the 
15  unique  pairs  of  the  6  stibscales.  The  number  of  times  a 
subscale  is  chosen  as  more  relevant  serves  as  the  subscale 
weight  for  that  operator.  A  TLX  score  from  0  to  100  is 
derived  by  multiplying  the  subscale  rating  for  a  task  by  the 
individual  subscale  weight,  summing  across  subscales,  and 
dividing  by  the  total  number  of  weights  (15) . 

The  three  forms  presented  in  Appendix  B  and  a  set  of 
cards  were  developed  from  NASA  Task  Load  Index  fTLX) :  Paoer- 
and-Pencll  Version  (1906) .  Pages  B-2  and  B-3  show  the 
instructions  that  aviators  were  given  on  the  use  of  the  NASA- 
TLX  rating  scales.  As  suggested  by  the  NASA  pamphlet,  the 
instructions  were  adapted  for  this  research.  Additionally, 
after  consulting  with  the  developer  of  the  scales,  the 
performance  scale  anchors  were  reversed  such  that  good 
appeared  at  the  right  of  the  scale  and  poor  appeared  on  the 
left;  preliminary  tests  with  the  scales  indicated  that  this 
scale  arrangement  corresponded  more  closely  with  raters' 
expectations . 

Page  B-4  presents  the  NASA  instructions,  entitled 
Soureefl  of  Workload.  These  instructions  were  used  with  a  set 
of  cards  to  collect  the  paired-comparison  data  needed  to 
generate  the  individual  scale  weights.  The  15  cards 
contained  all  possible  pairs  of  scale  titles. 

In  addition  to  the  TLX,  the  equally  weighted  mean 
subscale  workload  (MSW)  was  computed.  MSW  is  derived  by 
computing  the  mean  of  the  subscale  ratings.  The  MSW  is  a 
less  costly  measure  of  subjective  workload  because  the 
collection  and  use  of  the  paired-comparison  data  is  not 
required. 


Aviators 

Twenty  AH-64A  aviators  (10  battle-rostered  crews)  from 
one  battalion  of  an  operational  aviation  regiment 
participated  in  this  research.  All  crews  completed  all 
experimental  sessions. 
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The  demographic  data  for  the  20  aviators  Indicate  a 
range  of  experience  that  Is  typical  of  AH-64  operational 
units  (seo  Table  4) .  The  aviators  had  two  distinctly 
different  backgrounds:  those  with  previous  career  experience 
In  other  helicopters  (predominantly  the  AH-1)  and  those  who 
proceeded  directly  from  Initial  entry  rotary  wing  training  to 
the  AH-64  Aviator  Qualification  Course  (AQC) .  Across  all 
Indicators  of  experience,  the  pilots  were  more  experienced 
than  the  copilot/gunners. 

Apparatua 

Computer  teat  station.  The  computer  test  station  used 
to  present  the  conditions  to  the  aviators  consisted  of  an  IBM 
compatible  microcomputer  with  a  14-in  color  display  that  had 
a  phosphor  refresh  rate  of  60  Hz.  The  display  was  positioned 
approximately  one  meter  from  aviators'  eyes.  The  computer 
was  equipped  with  a  Systems  Research  Laboratory  Labpack  board 
that  performed  the  speech  synthesis  and  analog-to-digital 
sensing  of  the  tracking  controls.  The  computer  also  was 
equipped  with  a  separate  numeric  keypad  that  was  used  to 
collect  aviator  responses  during  dichotic  listening  trials. 

Physiological  recording  eqi:ipment .  Measures  of  aviator 
heart  rate  were  collected  using  small  silver/silver  chloride 


Table  4 

Aviator  Demographic  Data  at  the  Onset  of  the  Research 


Measure 

Pilot 
(n  -  10) 

Gunner 
(n  -  10) 

Combined 
(H  "  20) 

Age  (years) 

median 

range 

30 

(24-35) 

24.5 

(22-30) 

29 

(22-35) 

Months  on 
active  duty 

median 

range 

93.5 

(55-187) 

81 

(25-130) 

76.5 

(25-187) 

Months  since 

AQC 

median 

range 

39.5 

(12-56) 

17.5 

(0-28) 

26 

(0-56) 

AH-64A  flight 
hours 

median 

range 

462.5 

(250-900) 

230 

(30-390) 

300 

(30-900) 

Total  flight 
hours 

median 

range 

775 

(450-2300) 

479.5 

(190-600) 

505 

(190-2300) 

Note ■  AQC  -  Aviator  Qualification  Course. 
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(Ag/AgCl)  electrodes  (3M  Model  3320  Red  Dot)  and  amplified 
using  an  EKG  impedance  pneumograph  with  an  EKf?  amplifier  (UFI 
Model  RESP  1/EKG) .  To  ensure  good  signals,  the  electrode 
contact  impedance  was  measured  using  an  electrode  contact 
tester  (UFI  Model  1089  MK  II  Checktrode) . 

After  the  physiological  heart  signals  were  amplified, 
they  were  recorded  along  with  synchronization  signals 
generated  by  the  computers  using  an  8  channel  pulse-width- 
modulation  data  acquisition  system  (UFI  Model  3370/8) .  The  8 
channel  system  allowed  the  simultaneous  recording  of  the 
heart  signals  from  two  aviators  for  later  playback  and 
analysis . 


Procedures 

Tn-brlef ino .  Before  beginning  the  first  session,  the 
experimenter  briefed  the  participants  on  the  purpose  of  the 
research  and  on  the  experimental  procedures  and  tasks.  After 
the  briefing  was  completed,  the  aviators  were  instructed  to 
complete  the  demographic  questionnaires.  Copies  of  the  NASA- 
TLX  instructions  (see  pages  B-2  and  B-3)  were  given  to  each 
aviator.  The  aviators  were  instructed  to  read  the 
instructions  carefully  and  to  familiarize  themselves  with  the 
meaning  and  use  of  the  scales.  Finally,  the  aviators  were 
required  to  practice  rating  tasks  using  the  scales. 

Experiment  procedures.  Each  aviator  completed  three 
identical  experimental  sessions  over  a  six  day  period.  The 
sessions  were  separated  by  at  least  one  day.  When  the 
aviator  arrived  at  the  test  facility,  five  physiological 
recording  electrodes  were  applied  to  his  skin.  The  two 
electrodes  used  to  collect  heart  rate  signals  were  placed 
along  the  mid-axillary  lines  (left  and  right)  at  about  the 
level  of  the  rib  cage.  A  ground  electrode  was  placed  on  the 
abdomen.  The  aviator  was  then  seated  at  the  computer  test 
station,  the  electrode  leads  were  connected  to  the 
electrodes,  and  the  physiological  recording  was  initiated. 

Once  the  aviator  had  arranged  the  experimental  apparatus 
in  a  comfortable  configuration,  the  computer  program 
controlling  the  experiment  was  initiated.  The  program 
presented  each  aviator  with  three  trials  of  the  seven  tasks 
in  the  following  order:  xSxx,  Dxxx,  DSxx,  xSPx,  DSPx,  xSPT, 
DSPT.  Progress  through  the  21  trials  was  self  paced.  After 
each  3-minute  trial,  the  aviators  were  presented  with  a  brief 
description  of  the  upcoming  trial  and  were  required  to  press 
a  key  on  the  numeric  keypad  to  begin  the  next  trial. 

Appendix  C  shows  the  computer  file  that  controlled  the 
program  and  contained  the  aviator  instructions.  After  the 
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third  trial  of  each  condition,  the  aviators  were  Instructed 
to  perform  a  NASA-TLX  rating  on  the  completed  task. 

Dlchotlc  listening  performance  and  tracking  error  were 
collected  by  the  computer  during  all  trials.  Once  the 
aviator  had  completed  the  21  trials,  he  was  disconnected  from 
the  electrode  leads,  the  electrodes  were  removed,  and  he  was 
thanked  and  dismissed  for  the  day. 


Data  Analvaia  and-Sspring 

Tracking  performance.  The  computer  test  station 
computed  and  stored  tracking  performance  separately  for  each 
active  tracking  task.  During  each  trial,  the  computer 
measured  and  stored  the  distance  between  the  cursor  and  the 
target  In  centimeters  33  times  a  second.  After  each  trial, 
the  computer  calculated  the  root-mean-squared  (RMS)  error  for 
each  tracking  task. 

Past  research  has  shown  that  a  subject's  strategy  can 
have  dramatic  effect  on  the  performance  of  concurrent  tasks 
(Gopher,  1980;  Gopher,  Brlkner,  &  Navon,  1980;  Schneider  & 
Fisk,  1980;  Wlckens  &  Gopher,  1977) .  Some  subjects  may 
choose  to  devote  about  equal  attention  to  each  task  being 
performed  concurrently;  others  may  devote  far  more  attention 
to  one  task  than  to  another,  thereby  trading  poor  performance 
on  one  task  with  good  performance  on  another.  An  examination 
of  the  aviat  )rs'  tracking  performance  suggested  that  tracking 
performance  was.  Indeed,  Influenced  by  different  coping 
strategies.  Hence,  for  present  purposes,  it  was  not 
meaningful  to  examine  performance  separately  on  each  of  the 
three  tracking  tas^s.  Rather,  the  mean  RMS  error  for 
concurrent  tracking  tasks  was  computed  and  used  as  a  single 
Index  of  track!  ^  per*' )rmance .  For  example,  the  index  of 
tracking  perforn.''"’ce  on  condition  xSPx  was  computed  by 
summing  the  RMS  .  .jr  vhe  two-dimension  tracking  task  (S)  and 
the  horizontal  tracking  task  (P)  and  dividing  the  sum  by  2. 

In  the  case  of  the  xSPT  and  the  DSPT  conditions,  the  Index  of 
tracking  performance  was  computed  by  summing  mean  RMS  error 
for  all  three  tracking  tasks  and  dividing  by  3. 

Dlchotlc  Listening  performance .  During  the  DL  task, 
aviators  attended  to  the  digits  and  letters  presented  to  the 
designated  ear  and  attempted  to  type  the  9  digits  that 
occurred  in  each  DL  stimulus  string.  Three  measures  of  DL 
performance  were  calculated  by  the  computer  test  station: 
errors  of  commission,  omission,  and  exact  match.  Gopher  and 
Kahneman  (1971)  and  Gopher  (1982)  defined  errors  of 
commission  and  omission  and  used  the  measures  to  predict 
flying  proficiency  in  high  performance  aircraft.  Errors  of 
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omission  are  the  number  of  digits  present  in  the  DL  stimulus 
string  that  the  aviator  failed  to  enter.  Errors  of 
commission  are  the  number  of  entries  that  were  not  present  in 
the  DL  stimulus  string,  in  both  of  Gopher's  studies,  errors 
of  omission  were  the  best  predictors  of  success  in  flight 
training.  The  USAAVNC  uses  omission  errors  to  aid  in  the 
assignment  of  aviators  to  particular  aircraft  after  initial 
rotary  wing  training. 

Another  more  demanding  measure  of  DL  performance  was 
computed  for  each  trial .  An  exact  match  error  was  counted 
anytime  an  aviator's  input  deviated  from  the  DL  stimulus 
string.  The  aviators  that  participated  in  this  research  are 
recfuired  to  receive  numeric  strings  over  noisy  radios  during 
the  normal  conduct  of  their  flight  activities.  The  exact 
match  measure  was  collected  to  eliminate  the  possibility  of 
perfect  performance  (i.e.,  a  floor  effect)  for  errors  of 
omission  and  commission.  Additionally,  errors  of  exact  match 
can  be  computed  using  a  much  simpler  scoring  algorithm  than 
is  required  to  score  the  other  types  of  errors . 

interbeat  interval  data  correction.  The  heart  rate 
signals  were  analyzed  to  determine  the  series  of  interbeat 
intervals  of  the  heart  using  an  IBM-compatible  computer,  a 
Data  Translation  Model  2808  analog-to-digital  converter 
board,  and  the  workload  assessment  monitor  (WAM)  software 
developed  by  the  Human  Engineering  Group  of  the  Armstrong 
Aerospace  Medical  Research  Laboratory  at  Wright-Patterson  Air 
Force  Base,  Ohio.  Because  of  the  high  cost  of  analyzing 
heart  rate  variability  data,  only  the  data  compiled  on  Day  3 
were  analyzed.  The  WAM  software  was  used  to  analyze  the 
heart  rate  recordings  and  to  generate  data  files  containing 
discrete  lists  of  the  interbeat  intervals  (IBIs)  occurring 
during  each  trial. 

Review  of  the  output  data  files  generated  by  the 
software  indicated  that  it  had  missed  some  beats  in  most 
trials.  Missed  beats  are  indicated  by  a  jump  in  the  IBI 
interval  to  approximately  twice  the  average.  The  failure  of 
the  analysis  protocol  to  detect  every  heartbeat  is  not 
unusual  in  this  type  of  research.  Procedures  for  correcting 
missed  heartbeats  are  necessary  because  including  a  single 
IBI  with  twice  the  average  duration  has  substantial  effects 
on  estimates  of  the  variability  in  the  data. 

For  each  trial  that  had  missing  data,  the  data  were 
corrected  using  the  following  protocol.  A  running  mean  of 
the  previous  30  IBIs  was  calculated.  If  an  IBI  was  found 
that  was  1.6  times  the  current  mean,  the  IBI  was  divided  by 
the  integer  that  produced  corrected  IBIs  closest  to  the  IBI 
running  mean.  Consider  the  following  example.  The  IBI 
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running  mean  is  800.  An  IBI  of  2500  is  found  in  the  data 
file.  The  2500  IBI  is  divided  by  2,  3,  and  4  producing 
corrected  IBIs  of  1250,  833.33,  and  625,  respectively.  The 
2500  IBI  is  replaced  with  three  IBIs  of  833.33  because  they 
are  closest  to  the  current  running  mean  of  800.  If  WAM  did 
not  detect  a  beat  for  more  than  5  seconds  or  if  the  number  of 
missed  beats  exceeded  5%  of  the  data,  the  trial  was  excluded 
from  the  analysis .  The  heart  rate  data  for  two  aviators  were 
eliminated  from  analysis  because  the  number  of  trials 
rejected  was  too  large.  Of  the  remaining  18  aviators,  only  9 
of  the  378  trials  (18  aviators  by  7  conditions  by  3  trials) 
were  rejected;  in  no  case  was  more  than  one  of  the  three 
trials  for  a  condition  rejected. 

Time-based  measures  of  heart  rate  variability.  For  each 
of  the  18  aviators,  6  closely  related  time-based  measures  of 
heart  rate  variability  were  calculated  and  analyzed  using 
methods  described  by  Van  Oellen,  Aasman,  Mulder,  and  Mulder 
(1985) .  The  six  measures  are  listed  below: 

•  the  standard  deviation  of  the  IBI, 

•  the  coefficient  of  variation  of  the  IBI, 

•  the  root  mean  square  of  successive  Interval 
differences, 

•  the  coefficient  of  variation  of  successive  interval 
differences, 

•  the  sum  of  absolute  differences  between  successive 
IBIs,  and 

•  the  ratio  of  the  IBI  decelerations  to  the  number  of 
IBI  fluctuations. 

Although  the  heart  rate  variability  literature  indicates  that 
any  one  of  these  measures  would  be  sensitive  to  manipulation 
of  tas)c  demands,  each  was  computed  and  analyzed  for 
comparison  and  evaluation. 

Freauenev-based  measures  of  heart  rate  variability. 
Because  there  is  some  debate  about  the  relative  sensitivity 
of  time-based  versus  f reqpjency-based  measures  of  heart  rate 
variability  (Van  Dellen,  Aasman,  Mulder,  &  Mulder,  1985),  a 
subset  of  5  aviators'  data  was  transferred  to  the  Human 
Engineering  Group  of  the  Armstrong  Aerospace  Medical  Research 
Laboratory  at  Wright-Patterson  Air  Force  Base,  Ohio,  for 
further  analysis.  The  IBI  series  were  Fourier  analyzed  to 
measure  the  heart  rate  variability  in  two  frequency  bands. 

The  low  frequency  band  is  correlated  with  the  blood  pressure 
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control  system  and  is  centered  around  0.10  Hz.  The  high 
frequency  band  is  above  0.14  Hz.  Changes  in  the  high 
frequency  component  have  been  found  to  be  correlated  with 
changes  in  respiration  and  in  the  task  repetition  rate. 
Variability  at  the  characteristic  O.lO  Hz  frequency  for  the 
blood  pressure  band  corresponds  to  changes  in  the  IBI  series 
that  rise  and  fall  approximately  every  10  s;  variability  at 
the  characteristic  0.20  Hz  frequency  for  the  respiration  band 
corresponds  to  changes  in  the  IBI  series  that  rise  and  fall 
every  5  a.  These  two  frequency-based  measures  of  heart  rate 
variability  were  computed  using  the  MXEdit  software  developed 
by  Delta-Biometrics  and  the  results  were  returned  for 
subsequent  analysis  and  plotting. 

Statistical  analyses.  For  each  workload  measure,  a 
repeated-measures  analysis  of  variance  (ANOVA)  was  conducted 
to  test  its  sensitivity  to  manipulations  of  two  within- 
subjects  factors:  practice  (day)  and  task  demands 
(condition) .  The  ANOVA  assumption  of  homogeneity  of 
population  variance  probably  does  not  hold  for  distributions 
of  TLX  ratings,  RMS  tracking  error,  DL  performance,  or  heart 
rate  variability.  Although  violations  of  the  homogeneity  of 
variance  assumption  may  not  be  critical  with  completely 
randomized  designs,  it  can  seriously  affect  the 
interpretation  of  £  ratios  in  repeated-measures  designs.  In 
fact,  the  £  test  is  known  to  be  positively  biased  or  about  2 
or  3%  more  lenient  than  nominally  set. 

To  solve  the  problem  of  heterogeneity  of  variance  and  to 
maintain  the  consistency  of  the  tests  across  measures,  the 
Geisser-Greenhouse  correction  for  maximal  heterogeneity 
(Giesser  and  Greenhouse,  1958)  was  used.  The  method  assumes 
that  the  populations  are  not  homogeneous,  performs  the  usual 
analysis  of  variance,  and  evaluates  the  observed  £  ratios 
against  new  critical  values.  The  method  computes  the  £  ratio 
in  the  classical  manner  and  adjusts  the  p  values  assuming 
biased  populations.  When  main  effects  for  either  factor  were 
found,  post-hoc  one-way  ANOVAs  or  contrasts  were  performed 
between  adjacent  levels  of  the  factor  to  determine  the  level 
or  levels  where  change  occurred. 


Results 

The  following  four  subsections  describe  the  predictions 
generated  by  the  two  TAWL  models  and  the  results  of  the 
subjective,  performance,  and  physiological  workload  measures. 
The  two  final  subsections  describe  the  relationships  among 
these  variables. 
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TAWL  Workload  Predictions 


As  previously  described,  two  workload  prediction  models 
were  developed  using  the  TAWL  methodology.  The  following 
three  subsections  describe  the  structure  of  each  model  and 
compare  the  models'  structure  and  predictions. 

TAWT.  Model  1 .  TAWL  Model  1  has  seven  segments,  seven 
functions,  and  seven  tasks.  Appendix  D  presents  a  complete 
representation  of  Model  1.  Each  condition  was  modeled  using 
a  segment  decision  rule.  Four  functions  were  defined  to 
accomplish  the  tracking  tasks.  One  function  was  used  to 
control  each  of  the  three  tracking  tasks;  the  fourth  function 
was  used  to  visually  monitor  the  screen.  The  remaining  three 
functions  were  used  to  perform  the  DL  task.  One  function  was 
used  to  detect  the  beginning  of  the  audio  presentation;  one 
function  was  used  to  monitor  the  auditory  stimuli  during 
presentation;  and  one  function  was  used  to  enter  the 
responses  on  the  computer  keypad.  Seven  tasks  were  used  to 
construct  the  seven  functions.  One  task  was  defined  to 
accomplish  the  activities  in  each  of  the  functions  defined  in 
the  model . 

Table  5  presents  a  summary  of  the  workload  predictions 
generated  by  Model  1 .  Two  general  workload  metrics  are  shown 
in  Table  5:  the  number  of  overload  conditions  (OC)  and 
overload  density  (OD) .  Overload  conditions  were  predicted 
for  only  two  conditions;  DSPx  and  DSPT.  The  overload  density 
metric  Indicates  that  an  overload  condition  is  present  during 
40.8%  of  the  DSPx  condition  and  during  73.3%  of  the  DSPT 
condition.  Also  shown  in  Table  5  are  the  average  workload 
predictions  for  each  of  the  workload  components. 

TAWL  Model  2.  TAWL  Model  2  consists  of  seven  segments, 
eight  functions,  and  six  tasks.  Appendix  E  presents  a 
complete  representation  of  Model  2.  Each  condition  was 
modeled  using  a  segment  decision  rule.  He  defined  four 
functions  to  accomplish  the  tracking  tasks.  One  function  was 
used  to  control  each  of  the  three  tracking  tasks;  the  fourth 
function  was  used  to  visually  monitor  the  screen.  The  other 
four  functions  were  used  to  accomplish  the  DL  task;  one 
function  was  used  for  each  of  the  following: 

•  attend  to  the  auditory  stimuli  during  audio 
presentation, 

•  detect  the  beginning  of  the  audio  presentation, 

•  enter  the  responses  to  the  first  part  of  the  DL 
string,  and 
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Table  5 


Workload  Predictions  for  TAWL  Model  1 


Segment 

OC 

OD 

AUD 

KIN 

VIS 

COG 

PSY 

2: 

Stick  Only 

0 

0.0 

0.0 

1.0 

5.4 

2.4 

2.6 

1: 

DL  Only 

0 

0.0 

3.9 

0.4 

1.5 

4.6 

0.9 

3: 

DL  and  Stick 

0 

0.0 

3.9 

1.4 

4.7 

6.6 

3.5 

4: 

Stick  and  pedal 

0 

0.0 

0.0 

2.0 

5.4 

3.6 

5.2 

5: 

DL,  Stick,  and  Pedal 

64 

40.8 

3.9 

2.0 

4.7 

7.3 

5.0 

6: 

Stick,  Pedal,  and  Throttle 

0 

0.0 

0.0 

3.0 

5.4 

4.8 

7.8 

7: 

DL,  Stick,  Pedal,  and  Throttle 

126 

73.3 

3.9 

3.0 

4.7 

8.5 

7.6 

Wota.  Th«  order  of  the  segments  was  changed  to  correspond  with  the 


order  of  task  presentation  in  the  ejqperimsnt  and  with  the  order  used  in 
the  other  model.  The  following  abbreviations  are  used:  OC  Overload 
Condition,  OD  >  Overload  Density  expressed  as  a  percent  of  total  segment 
time,  AUO  ■  Auditory,  KIN  >  Kinesthetic,  VIS  >  visual,  COG  «  Cognitive, 
PSY  Psychofflotor,  Dt  ■  Dichotic  Listening. 


•  enter  the  responses  to  the  second  part  of  the  DL 
string. 

Six  tasks  were  used  to  construct  the  eight  functions. 

One  task  controlled  the  vertical  and  one  task  controlled  the 
horizontal  position  of  the  cursors  in  the  tracking  functions. 
In  addition,  one  task  wa.s  defined  to  perform  each  of  the 
following: 

•  monitor  the  computer  screen, 

•  enter  one  DL  response, 

•  monitor  the  DL  stimuli  during  string  presentation,  and 

•  detect  the  beginning  of  DL  string  presentation. 

A  summary  of  the  workload  predictions  generated  by  the 
Model  2  is  shown  in  Table  6.  Overload  conditions  were 
predicted  only  for  the  DSPT  condition;  an  overload  condition 
is  present  28.9%  of  the  time. 

Comparison  of  TAWL  models.  Although  the  workload 
prediction  models  developed  by  the  two  analysts  were  similar, 
the  models  differed  in  two  ways.  First,  there  were 
differences  in  the  structure  of  the  two  models.  The  top-down 
analysis  of  the  conditions  produced  two  different  functional 
descriptions  of  the  situation.  These  differences  are  not 
judged  to  be  particularly  important  because,  with  one 
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Table  6 


Workload  Predictions  for  TAWL  Model  2 


Segment 

oc 

OD 

AUD 

KIN 

VIS 

COG 

PSY 

1:  stick  (ST) 

0 

0.0 

0.0 

NR 

5.4 

1.0 

2.6 

2:  Dichotic  Listening  (DL) 

0 

0.0 

3.6 

NR 

1.4 

3.4 

0.6 

3:  ST  and  DL 

0 

mSi 

3.6 

NR 

5.3 

4.4 

3.2 

4;  ST  and  Pedals  (PD) 

0 

BBl 

0.0 

NR 

5.4 

2.0 

5.2 

5:  ST,  DL,  and  PD 

0 

3.6 

NR 

5.3 

5.4 

5.8 

6:  ST,  PD,  and  Throttle  (TR) 

0 

■nf:l 

0.0 

NR 

5.4 

3.0 

7.8 

7:  ST,  DL,  PD,  and  TR 

3.6 

NR 

5.3 

6.4 

8.4 

Mote.  The  following  abbreviations  are  used:  OC  -  Overload  Condition, 
OD  «  Overload  Density,  AUD  -  Auditory,  KIN  Kinesthetic,  VIS  •  Visual, 
COO  -  Cognitive,  PSY  *  Psychomotor,  NR  >  Not  Rated. 


exception,  both  models  produced  the  same  sequence  of  tasks. 
The  exception  concerned  the  status  of  the  tracking  tasks 
during  the  numeric  entry  for  the  dichotic  listening  task. 

One  analyst  interrupted  (stopped  the  performance  of)  the 
tracking  tasks  during  numeric  entry;  the  other  analyst  did 
not.  The  other  differences  between  the  models  reflect  subtle 
differences  in  the  way  the  analysts  performed  the  top-'down 
analysis.  Differences  of  this  type  Illustrate  that  the  TOSS 
software  provides  alternative  methods  for  constructing 
functionally  equivalent  models.  Most  of  the  structural 
differences  between  the  models  did  not  contribute  to 
differences  in  the  models'  predictions.  However,  the 
predictions  were  influenced  by  whether  or  not  the  model 
assumed  concurrent  performance  of  the  tracking  tasks  and 
numeric  entry  tasks.  In  fact,  the  difference  in  assumptions 
about  concurrent  performance  of  tracking  and  numeric  entry 
accounts  entirely  for  the  different  number  of  psychomotor 
overloads  predicted  by  the  models. 

The  second  way  in  which  the  models  differed  was  in  the 
analysts'  workload  ratings.  For  exan^le,  one  analyst  matched 
the  cognitive  component  of  the  tracking  tasks  with  the 
workload  rating  scale  anchor  "Alternative  Selection"  (1.2) 
and  the  other  analyst  matched  it  with  "Simple  Association" 
(1.0).  Additionally,  one  analyst  rated  the  cognitive 
component  of  the  dichotic  listening  task  with  the  scale 
anchor  ''Encoding/Decoding,  Recall"  (5.3)  and  the  other 
analyst  matched  it  with  "Evaluation/ Judgment  (Consider  Single 
Aspect)"  (4.6).  The  differences  in  workload  ratings  account 
for  the  high  number  of  cognitive  overloads  in  Model  1 
(segments  6  and  7),  relative  to  Model  2. 
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The  relationship  between  the  two  models  was  assessed  In 
two  steps.  First,  a  Pearson  product -moment  correlation 
coefficient  (x)  was  calculated  for  each  workload  component. 
The  X  values  for  the  auditory,  visual,  cognitive,  and 
psychomotor  predictions  were  1.00,  .98,  .99,  and  .99, 
respectively.  Second,  an  overall  workload  prediction  for 
each  condition  was  derived  by  averaging  the  workload  across 
components  (see  Figure  2) .  The  x  value  for  the  overall 
workload  predictions  from  the  models  was  .99.  The  fact  that 
the  coefficients  are  so  close  to  unity  indicates  that  the 
predictions  generated  by  the  two  models  are  highly  related 
despite  the  differences  in  the  models'  structure. 


Subjective  Ratings 

For  each  experimental  session,  aviators  completed  the 
NASA-TLX  rating  forms  after  finishing  the  third  trial  of  each 
condition.  The  individually  weighted  sum  of  subscales  (the 
TLX)  and  the  equally  weighted  mean  subscale  workload  (MSW) 
were  analyzed.  Analyses  to  compare  the  TLX  and  the  MSW  were 
conducted  to  evaluate  the  claim  of  Byers,  Bittner,  and  Hill 
(1989)  that  the  more  easily  computed  MSW  is  nearly  as 
effective  as  the  TLX  for  assessing  subjective  workload, 

Subseale  analysis.  Before  the  NASA-TLX  could  be  used  as 
a  measure  of  TAWL  validity,  some  analysis  of  the  ratings' 
sensitivity  to  task  demands  was  necessary.  Additionally,  the 
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Figure  2 .  Overall  predicted  workload  for 
Models  1  and  2  as  a  function  of  condition. 
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subscale  results  needed  to  be  examined  to  confirm  that  the 
aviators  used  the  scales  appropriately.  To  begin  these 
analyses,  the  NASA-TLX  subscale  ratings  were  pooled  across 
aviators,  means  were  calculated,  and  the  means  were  plotted 
as  a  function  of  day  and  condition.  As  can  be  seen  in  Figure 
3,  ratings  for  most  subscales  varied  systematically  as  a 
function  of  both  day  and  condition.  With  only  one  exception 
(own  performance),  the  ratings  decreased  as  a  function  of  day 
and  Increased  as  a  function  of  condition.  In  contrast,  the 
ratings  of  own  performance  did  not  vary  as  a  function  of  day 
but  decreased  as  a  function  of  condition. 

The  error  bars  in  Figure  3  depict  the  size  of  the 
standard  error  (plus  and  minus)  of  each  mean.  The  error  bars 
are  presented  in  Figure  3  and  in  subsequent  figures  simply  to 
provide  an  indication  of  the  across-aviator  variability  in 
each  data  set . 

An  indication  of  the  sensitivity  and  appropriate  use  of 
the  subscales  can  be  seen  in  the  plot  of  physical  demand  (see 
Figure  3) .  The  graph  shows  that  the  physical  demand  rating 
of  the  dichotic  listening  only  (Dxxx)  condition  is 
considerably  lower  than  the  ratings  for  the  other  conditions. 
Presumably,  this  reflects  the  low  physical  demand  of  this 
condition  relative  to  the  other  multi-tas)c  conditions  and  the 
single  tracking  task  condition  (xSxx) .  Without  exception, 
the  relationships  between  the  NASA>TLX  subscale  values  and 
both  day  and  condition  conform  with  expectations. 

A  repeated-measures  ANOVA  was  conducted  to  test  the 
sensitivity  of  each  of  the  subscales  to  practice  (day)  and 
task  demands  (condition) .  Table  7  presents  the  results  of 
the  18  E  tests.  Although  some  of  the  tests  were  significant 
and  some  were  not,  the  Geisser-Greenhouse  correction  did  not 
change  the  significance  of  any  of  the  tests  from  the  nominal 
£  criteria.  The  effect  of  day  was  significant  for  all 
subscales  except  own  performance;  the  effect  of  condition  was 
significant  for  all  subscales.  A  significant  day  by 
condition  interaction  was  found  for  two  subscales:  mental 
demand  and  physical  demand.  The  significant  Interactions 
indicate  that  the  effect  of  practice  differed  as  task  demands 
changed. 
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Condition 


Figure  3.  Ratings  on  six  NASA-TLX  subscales  as  a  function  of 
day  and  condition.  (The  error  bars  represent  ±  1  standard 
error  of  the  mean . ) 


Task  load  index  and  mean  subsea le  workload  analysis. 

With  evidence  that  the  aviators  used  the  subscales 
appropriately  and  that  the  subscales  were  sensitive  to  the 
experimental  manipulations,  the  NASA-TLX  and  the  MSW  were 
computed  and  plotted  (see  Figure  4) .  As  can  be  seen  in 
Figure  4,  the  ratings  varied  systematically  as  a  function  of 
both  day  and  condition;  the  ratings  decreased  as  a  function 
of  day  and  increased  as  a  function  of  condition. 

Repeated-measures  ANOVA  tests  were  conducted  to  test  TLX 
and  MSW  sensitivity  to  practice  (day)  and  task  demands 
(condition) .  The  last  two  rows  Table  7  present  the  results 
of  the  6  E  tests.  Both  measures  showed  significant 
differences  as  a  function  of  day  and  condition  and  the  TLX 
measure  showed  a  significant  interaction  between  day  and 
condition.  However,  when  the  Geisser-Greenhouse  correction 
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Table  7 

£  Ratios  Resulting  From  ANOVA  Tests  of  NASA-TLX 
Sensitivity  to  Day  and  Condition 


Variable 

Day 

Conditions 

Day  by  condition 

Scale 

df  -  2,38 

df  -  6,114 

df  -  12,228 

MD 

26.79** 

78.64** 

3.76** 

PD 

13.46** 

120.39** 

3.03** 

TD 

16.15** 

52.32** 

1.17 

OP 

2.50 

7.01** 

0.75 

EF 

15.05** 

24.61** 

1.00 

FR 

11.25** 

73.28** 

1.45 

TLX 

23.73** 

95.57** 

2.84* 

MSW 

23.03** 

104.19** 

2.22 

Nqf.e.  The  following  abbreviations  are  used  as  column  headings;  MD  > 
Mental  Demand,  TO  -  Temporal  Demand,  OD  -  Own  Performance,  EP  - 
Effort,  FR  *  Frustration,  TLX  *  Task  Load  Index,  and  MSN  >  Mean 
Subscale  Workload. 

*Ad justed  values  of  <  .05.  **Ad justed  values  of  p  <  .01. 


was  used,  the  interaction  was  not  significant  for  the  MSW. 

In  summary,  the  results  indicate  that  both  measures  decrease 
significantly  as  a  function  of  practice  and  increase 
significantly  as  a  function  of  task  demand. 

Scale  relationships.  Pearson  product -moment  correlation 
coefficients  (x)  were  computed  to  assess  the  relationships 
among  the  subscales,  the  TLX,  and  the  MSW.  Only  the  data  for 
the  final  day  of  practice  were  analyzed  for  this  purpose. 
Table  8  shows  the  results  of  the  analyses.  The  correlations 
among  five  of  the  six  subscales  were  found  to  be  high  (.42  S 
X.  ^  .83)  and  positive.  In  contrast,  the  relationship 
between  own  performance  and  all  other  subscales  was  found  to 
be  either  negative  and  low  (.13  S  x  S  .30)  or  not 
statistically  significant.  These  findings  are  logically 
consistent . 

Specifically,  the  findings  show  that  (a)  conditions  that 
are  demanding  with  respect  to  one  dimension  (e.g.,  mental 
demand)  are  also  demanding  with  respect  to  other  dimensions 
(e.g.,  physical  demand  and  temporal  demand),  (b)  conditions 
that  are  the  most  demanding  tend  to  require  the  most  effort 
and  create  the  most  frustration,  (c)  assessments  of  own 
performance  tend  to  decrease  as  task  demands  increase,  and 
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Figure  4 .  The  Task  Load  Index  and  mean  subscale 
workload  as  a  function  of  day  and  condition.  (The 
error  bars  represent  ±  1  standard  error  of  the  mean.) 


(d)  as  task  demand  increases^  greater  effort  does  not 
necessarily  result  in  better  performance. 

The  correlation  between  the  individually  weighted  TLX 
and  the  equally  weighted  MSW  was  found  to  be  positive  and 
very  high  (£  •  .96).  The  powerful  relationship  between  the 
measures  strongly  supports  Byers'  claim  that  MSW  is  nearly  as 
effective  as  TLX  for  assessing  subjective  workload  (Byers  et 
al.,  1989).  The  similarity  of  the  two  measures  also  is 
indicated  by  the  similarity  in  the  strength  of  their 


Table  8 


Correlations  Among  the  NASA-TLX  Subscale,  the  TLX,  and  the 
MSW  Ratings 


Scale  MD  PD  TD  OP  EF  FR  TLX 


Kotie.  For  all  correlations  U  >  420.  The  following  abbreviations  are 
used  as  column  headings :  MD  *  Mental  Demand,  TD  -  Temporal  Demand,  OP  - 
Own  Performance,  EF  Effort,  FR  ■  Frustration,  TLX  *  Task  Load  Index, 
TLX  -  Task  Load  Index,  and  MSW  >  Mean  Subscale  Workload. 

<  .01. 


relationships  with  the  individual  subscales.  Table  8  shows 
that,  for  five  subscales,  the  subscales'  correlation  with  TLX 
is  nearly  Identical  to  the  corresponding  subscales' 
correlation  with  MSW,*  the  correlation  coefficients  for 
corresponding  subscales  differ  by  ±.05  or  less.  The 
correlation  between  own  performance  and  TLX  is  negative  and 
statistically  significant  (t  «»  -.19);  the  correlation  between 
own  performance  and  MSW  also  is  negative  but  is  not  large 
enough  to  reach  statistical  significance  (x  •  -.02). 

The  relationships  quantified  in  Figures  3  and  4  and  in 
Tables  6  and  7  provide  convincing  evidence  that  the  aviators 
who  participated  in  this  study  used  the  TLX  rating  scales  as 
they  were  designed  to  be  used,  and  that  the  resulting  ratings 
provide  a  sensitive  measure  of  workload  that  can  be  used  as  a 
criterion  in  validating  the  TAWL  methodology. 


Performance  Measures 

The  computer  test  station  produced  objective  measures  of 
tracking  and  dichotlc  listening  performance.  These  two 
measures  were  analyzed  to  confirm  that  they  were,  in  fact, 
sufficiently  reliable  and  sensitive  to  be  used  as  criteria 
for  validating  the  TAWL  methodology.  The  results  of  the 
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performance  measures  analyses  are  presented  in  the  following 
two  subsections. 

Tracking  performance.  Figure  5  shows  RMS  error  as  a 
function  of  day  and  condition.  The  curves  in  Figure  5 
support  the  assumption  that  the  task  conditions  employed  in 
this  study  present  different  demands  on  the  aviators.  It  can 
be  seen  in  Figure  5  that  (a)  RMS  is  higher  for  the  two-task 
condition  (DSxx)  than  for  the  single-task  condition  (xSxx) , 

(b)  both  three-task  conditions  have  higher  RMS  error  than 
either  two-task  condition,  and  (c)  the  four-task  condition 
has  higher  RMS  error  than  either  three-task  condition. 

The  size  of  the  increment  in  RMS  error  that  results  from 
adding  a  single  task  depends  on  the  task  that  is  added. 

Adding  a  second  tracking  task  (xSxx  vs.  xSPx)  results  in  a 
larger  Increment  in  RMS  error  than  adding  the  dichotic 
listening  task  (xSxx  vs.  DSxx).  However,  adding  a  third 
tracking  task  results  in  about  the  same  increment  in  RMS 
error  as  adding  a  DL  task  to  the  same  two-task  condition. 

The  largest  increment  in  RMS  error  results  from  adding  a 
fourth  task  to  a  three-task  condition;  the  addition  of  a 
third  tracking  task  to  Condition  DSPx  results  in  about  the 
same  increment  in  error  as  adding  the  dichotic  listening  task 
to  Condition  xSPT.  Despite  the  attempt  to  use  the  four-task 
condition  (DSPT)  to  study  operator  strategies  under  overload 


Figure  5.  Tracking  performance  as  a  function 
of  day  and  condition.  (The  error  bars 
represent  ±  1  standard  error  of  the  mean.) 


30 


conditions,  performance  on  that  condition  did  not  degrade 
catastrophically  as  was  expected. 

A  comparison  of  the  RMS  error  curves  for  the  three  days 
indicates  that  although  practice  on  Day  1  resulted  in  a 
substantial  decrease  in  error  on  Day  2,  no  further 
improvement  in  performance  resulted  from  the  practice  on  Day 
2.  It  is  important  to  note,  however,  that  the  relationships 
between  RMS  error  and  condition  described  above  did  not 
change  with  practice  on  either  Day  1  or  Day  2. 

Repeated-measures  ANOVA  tests  were  conducted  to  test  the 
effect  of  practice  (day)  and  task  demands  (condition)  on 
tracking  performance.  Practice  effected  tracking  performance 
significantly,  £  (2,  38)  ■  41.03/  ji  <  .0001;  the  difference 
between  Day  1  and  Day  2  was  significant  but  the  difference 
between  Day  2  and  Day  3  was  not.  Task  demands  also  effected 
tracking  performance  significantly,  Z  (5,  95)  -  103.09,  a  < 
.0001.  With  only  one  exception,  conditions  were  different 
from  the  preceding  ones;  condition  DSPx  did  not  differ 
significantly  from  condition  xSPT. 

The  aviators  employed  different  strategies  to  cope  with 
the  task  demands  (both  physical  and  mental)  of  the  four-task 
condition  (DSPT) .  All  20  twenty  aviators  employed  some 
strategy  to  maintain  performance;  none  were  observed  to 
ignore  or  shed  one  of  the  tasks.  The  two  most  common 
strategies  were  (a)  to  control  the  vertical  tracking  joystick 
with  the  left  forearm  while  entering  DL  responses  with  the 
left  hand,  and  (b)  to  control  the  vertical  tracking  joystick 
with  the  left  hand,  the  dual-axis  joystick  with  the  right 
hand,  and  enter  the  DL  responses  with  the  right  hand 
intermittently. 

DL  performance.  The  dichotic  listening  task  was 
included  in  four  of  the  seven  conditions  used  in  the 
research.  Figure  6  presents  the  number  of  DL  errors  of 
commission,  omission,  and  exact  match  as  a  function  of  day 
and  condition;  in  all  cases,  errors  were  averaged  across 
aviators.  As  can  be  seen  in  Figure  6,  a  floor  effect  was  not 
evident  in  any  of  the  data;  all  three  measures  showed  higher 
error  rates  as  task  demand  increased.  The  errors  of 
commission  were  less  orderly  and  more  variable  than  either 
omission  or  exact  match  errors.  The  trends  for  omission  and 
exact  match  errors  appear  quite  similar. 

Other  studies  of  dichotic  listening  have  found  dichotic 
listening  performance  to  be  better  with  the  left  than  the 
right  ear  (Bryden,  1969;  Gopher  &  Kahneman,  1971;  Kimura, 

1967;  Treisman  &  Geffen,  1968) .  Error  rates  were  higher  when 
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Dichotic  Listening  Errors 


Dxxx  DSxx  DSPx  DSPT 

Condition 

Figure  6.  Dichotic  listening  performance  as  a 
function  of  day  and  condition.  (The  error  bars 
represent  ±  1  standard  error  of  the  mean . ) 
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the  relevant  stimuli  were  presented  to  the  left  ear.  The 
mean  per-stimulus  error  rates  for  errors  of  commission, 
omission,  and  exact  match  for  the  left  and  right  ear  were  .40 
(L)  vs.  .34  (R),  .62  (L)  vs.  .45  <R) ,  and  1.67  (L)  vs.  1.39 
(R) ,  respectively. 

Three  repeated-measures  ANOVA  tests  were  conducted  to 
test  the  effect  of  practice  (day)  and  task  demand  (condition) 
on  DL  performance.  Table  9  shows  the  size  and  statistical 
significance  of  the  9  E  ratios  generated  by  the  ANOVA  tests. 
The  effect  of  day  was  significant  for  all  measures  except 
errors  of  commission.  The  effect  of  condition  was 
significant  for  all  measures.  None  of  the  measures  showed  a 
significant  day  by  condition  interaction.  For  both  omission 
and  exact  match  error,  the  difference  between  Day  1  and  Day  2 
was  significant  but  the  difference  between  Day  2  and  Day  3 
was  not . 

In  summary,  both  objective  measures  used  in  this 
research  indicate  that  (a)  practice  after  Day  1  did  not 
result  in  improved  aviator  performance,  and  (b)  error 
increases  systematically  as  a  function  of  task  demand 
throughout  the  range  of  conditions  used. 

Physiological  Measures 

Eight  time-  and  frequency-based  measures  of  heart  rate 
variability  were  collected  for  each  experimental  trial.  The 
measures'  sensitivity  to  manipulations  of  task  demand  were 
analyzed  to  determine  their  utility  as  criteria  for 
validating  the  TAWL  methodology.  The  results  of  the  analyses 
are  presented  in  the  following  three  subsections. 


Table  9 

F  Ratios  Resulting  From  ANOVAs  Performed  to  Test  the  Effect 
of  Day  and  Condition  on  Dichotic  Listening  Performance 


Error  Measure 

Variable 

Day 

SLL  -  2,38 

Conditions 
df  -  3,57 

Day  by  condition 
df  -  6,114 

Commission 

1.59 

1.82** 

2.01 

Omission 

4.62* 

19.47** 

1.49 

Exact  Match 

8.62** 

27.99** 

1.60 

^Adjusted  values 

of  p  <  .05.  *‘ 

Adjusted  values  of 

p  <  .01. 
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Time-based  measures.  The  analysis  of  heart  rate  and 
heart  rate  variability  was  begun  by  computing  each  aviator' s 
average  IBI  for  each  condition.  Then  the  mean  IBIs  were 
pooled  across  aviators  and  a  grand  mean  IBI  was  calculated 
for  each  condition;  the  grand  mean  IBI  for  each  of  the  seven 
conditions  Is  shown  In  Figure  7.  As  can  be  seen  In  Figure  7, 
the  IBIs  were  longest  for  the  xSxx  condition;  shorter  for  the 
Dxxx  and  DSxx  conditions;  and  shortest  for  the  xSPx,  DSPx 
xSPT,  and  DSPT  conditions. 

A  repeated-measures  ANOVA  showed  a  significant  effect 
for  condition,  £  (6,  102)  -  10.60,  p  <  .0001;  however,  the 
only  significant  difference  between  successive  points  was 
between  the  DSxx  condition  and  xSPx  condition.  IBI,  In 
Itself,  was  not  expected  to  be  sensitive  to  manipulations  of 
task  demands;  the  measure  is  presented  here  to  aid  In  the 
interpretation  of  heart  rate  variability  described  in  the 
following  paragraphs. 

The  six  time-based  measures  of  heart  rate  variability 
described  in  the  methods  section  were  computed,  analyzed,  and 
plotted  as  a  function  of  condition.  For  ease  of  comparison, 
the  data  for  all  six  measures  are  shown  in  Figure  8. 
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Figure  7.  Mean  heart  interbeat  interval  as 
a  function  of  condition,  (The  error  bars 
represent  ±  1  standard  error  of  the  mean.) 
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CondMon  Common 


Figure  8.  Six  measures  of  heart  rate  variability  as  a 
function  of  condition.  (The  error  bars  represent  ±  1 
standard  error  of  the  mean.) 


The  data  in  Figure  8  must  be  interpreted  in  light  of  the 
fact  that  measures  of  heart  rate  variability  are  expected  to 
decrease  as  tas)c  demand  increases  (Aasman  et  al.,  1987).  An 
examination  of  the  curves  in  Figure  8  shows  that  none  of  the 
6  time-based  measures  of  heart  rate  variability  conform  with 
expectations.  To  the  contrary,  the  plots  of  four  measures 
suggest  that  heart  rate  variability  increases  as  task  demand 
increases . 
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Figure  8  shows  that  condition  Dxxx  produced  a  very  high 
degree  of  heart  rate  variability  relative  to  most  other 
conditions.  For  two  measures  (IBI  Standard  Deviation  and  IBI 
Coefficient  of  Variation),  heart  rate  variability  was  higher 
for' condition  Dxxx  than  for  any  other  condition;  for  the 
remaining  four  measures,  heart  rate  variability  was  higher 
for  condition  Dxxx  than  for  most  other  conditions. 

A  repeated-measures  ANOVA  was  conducted  for  each  of  the 
six  time-based  measures  of  heart  rate  variability.  A 
significant  condition  effect  was  found  only  for  IBI  standard 
deviation,  £  <6,  102)  -  2.36,  p  <  .05,  and  IBI  coefficient  of 
variation,  £  (6,  102)  -  2.55,  p  <  .05.  For  both  measures, 
the  Dxxx  condition  differed  from  the  xSxx  condition  and  DSxx 
condition  but  no  other  successive  contrast  was  significant . 
Hence,  although  two  measures  of  heart  rate  variability  vary 
significantly  as  a  function  of  condition,  the  relationship  is 
opposite  to  the  findings  in  the  literature.  The  cause  of  the 
unexpected  relationships  is  unknown. 

Frequenoy-baaed  meafliirea.  The  first  Step  of  the 
frequency  analysis  of  heart  rate  variability  eliminates 
approximately  20  IBIs  from  the  beginning  and  end  of  each 
trial.  Mean  IBI  was  calculated  for  the  remaining  data  and  is 
presented  in  Figure  9  to  aid  in  the  interpretation  of  the 
measures  of  heart  rate  variability  described  in  the  following 
paragraphs . 


Condition 


Figure  9.  Heart  interbeat  interval  as  a 
function  of  condition.  (The  error  bars 
represent  ±  1  standard  error  of  the  mean.) 
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The  frequency-based  analysis  measured  the  variability  of 
the  IBI  series  in  two  frequency  bands.  The  low  frequency  or 
blood  pressure  band  ranged  from  0.06  and  0.14  Hz.  The  high 
frequency  band  or  respiration  band  ranged  from  0.15  to  0.40 
Hz.  Figure  10  presents  the  results  of  these  analyses.  In 
general/  the  trends  for  both  measures  were  similar.  The 
respiration  band  had  a  greater  amount  of  variance  than  the 
blood  pressure  band.  Both  measures  showed  high  heart  rate 
variability  for  the  Dxxx  condition.  However,  the  effects  of 
condition  were  not  significant  for  either  of  the  measures . 

Summary .  Most  time-  and  frequency-based  measures  of 
heart  rate  variability  were  not  sensitive  to  manipulations  in 
task  demands.  When  the  measures  showed  significant 
differences  between  conditions,  the  trends  in  the  data  were 
clearly  different  from  the  trends  apparent  in  the  performance 
and  subjective  measures.  For  these  reasons,  measures  of 
heart  rate  variability  were  judged  to  be  of  little  value  as 
criteria  for  validating  the  TAWL  methodology.  Although, 
heart  rate  variability  is  not  discussed  further  in  the  main 
body  of  this  report,  interested  readers  can  find  the  heart 
rate  variability  statistics  in  the  Appendix  F. 


Figure  10.  Two  frequency-based  measures  of 
heart  rate  variability  as  a  function  of 
condition.  (The  error  bars  represent  ±  1 
standard  error  of  the  mean.) 
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Relationships  Among  Measures 


The  relationships  between  individual  measures  and 
conditions  (task  demands) ,  described  in  previous  sections, 
are  the  relationships  that  have  the  greatest  bearing  on  the 
selection  of  the  measures  that  are  to  serve  as  criteria  for 
assessing  the  validity  of  the  TAVJL  methodology.  However,  it 
also  is  worthwhile  to  examine  the  relationship  among  the 
measures  themselves.  It  is  of  particular  interest  to 
determine  the  intercorrelation  among  the  measures  that  were 
found  individually  to  be  highly  correlated  with  task  demand. 

A  low  intercorrelation  among  measures  (that  are  each  highly 
correlated  with  task  demand)  would  suggest  that  task  demand 
is  multidimensional  and  that  the  measures  serve  to  quantify 
two  or  more  of  the  different  dimensions.  Conversely,  high 
intercorrelations  among  measures  indicates  that  task  demand, 
as  defined  in  this  study,  is  unidimensional  and  can  be 
quantified  effectively  with  several  different  measures. 

Appendix  F  is  a  comprehensive  intercorrelation  matrix 
that  shows  the  Pearson  product -moment  correlation  coefficient 
(x)  between  every  possible  pair  of  measures  investigated  in 
this  study.  Although  many  of  the  correlation  coefficients 
shown  in  Appendix  F  have  no  direct  relevance  for  this  study, 
a  comprehensive  intercorrelation  matrix  was  developed  with 
the  expectation  that  the  data  may  be  of  value  to  other 
researchers  who  have  an  interest  in  workload  measurement. 

All  correlation  coefficients  shown  in  Appendix  F  and  all 
correlation  coefficients  discussed  below  were  computed  using 
only  the  data  compiled  during  the  last  practice  day  because 
aviators'  proficiency  was  highest  and  their  performance  most 
stable  on  the  final  day. 

Table  10  shows  the  intercorrelation  among  five  measures 
that,  individually,  were  highly  correlated  with  task  demand. 
It  can  be  seen  in  Table  10  that  errors  of  commission  is  the 
only  measure  that  was  not  found  to  be  highly  correlated  (x  i 
.985)  with  every  other  measure.  Although  the  correlation 
coefficients  between  errors  of  commission  and  the  other  four 
measures  varied  from  .789  to  .912,  they  were  not  large  enough 
to  reach  statistical  significance  at  the  .05  level  because  of 
the  small  number  of  pairs  in  the  data  set  (a  -  3  or  4) .  The 
high  correlation  among  RMS  tracking  error,  DL  omission 
errors,  DL  exact  match  errors,  and  TLX  leave  little  doubt 
that  these  measures  are  assessing  the  same  thing.  It  is 
possible  that  dichotic  listening  errors  of  commission  measure 
a  different  dimension  than  the  other  four  measures.  However, 
a  more  likely  explanation  is  that  errors  of  commission  are  a 
less  effective  measure  than  the  other  four. 
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Table  10 


Correlation  Coefficients  Among  Effective  Workload  Measures 


Measure 

RMS 

OMISS 

£ 

.996 

.053 

3 

COMISS 

£ 

.789 

£ 

.422 

XL 

3 

EXERR 

£ 

.996 

SL 

.059 

£ 

3 

TLX 

£ 

.990 

.000 

XL 

6 

MSW 

£ 

.985 

.000 

XI 

6 

OMISS  COMISS  EXERR  TLX 


MQta.  The  following  abbreviations  are  used  as  headings:  RMS  •  Root  Mean 
Squared  Tracking  Error,  OHZSS  -  Oichotic  Listening  (DL)  Errors  of 
omission,  COMISS  -  OL  Errors  of  Commission,  EXERR  ■  DL  Exact  Match 
Errors,  TLX  -  NASA  Task  Load  Index,  and  MSN  •  Mean  Subscale  workload. 


Relationship  Between  TAWL  Predictions  and 

Critsxion  Meaaucga 

The  results  presented  previously  show  that  the  TAWL 
models  generated  workload  predictions  that  varied 
systematically  and  in  an  expected  manner  with  the  seven 
conditions  presumed  to  represent  different  levels  of  task 
demand.  Furthermore,  results  were  presented  showing  that 
both  subjective  workload  measures  and  performance  measures 
(but  not  physiological  measures)  varied  systematically  and  in 
an  expected  manner  with  conditions.  Given  these  results,  it 
is  meaningful  to  assess  the  validity  of  the  TAWL  methodology 
by  determining  the  extent  to  which  the  TAWL  predictions  are 
related  to  the  subjective  and  performance  measures  that  have 
been  shown  to  be  highly  correlated  with  conditions. 
Specifically,  it  is  assumed  that  statistically  significant 
correlations  between  the  TAWL  predictions  and  the  measures 
used  as  validation  criteria  supports  the  conclusions  that  the 
TAWL  methodology  generates  valid  predictions  of  operator 
workload. 
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The  relationship  among  TAWL  predictions  and  selected 
criterion  measures  are  shown  in  Table  11.  Correlation 
coefficients  are  shown  for  all  but  one  of  the  combinations 
of  (a)  6  TAWL  component  predictions  (visual,  auditory, 
kinesthetic,  cognitive,  and  psychomotor)  and  the  average  of 
the  component  predictions,  and  (b)  two  performance  measures 
(RMS  tracking  error,  DL  exact  match  error)  and  one 
subjective  rating  measure  (TLX  workload  index) .  Because 
the  TAWL  predictions  for  the  auditory  component  were 
constant  for  all  DL  conditions,  it  was  not  possible  to 
compute  a  correlation  coefficient  between  the  TAWL  auditory 
component  predictions  and  DL  exact  match  errors.  Errors  of 
commission  was  excluded  as  a  criterion  measure  because  it 
was  judged  to  be  less  reliable  than  the  other  two  measures 
of  dichotic  listening.  Errors  of  omission  and  TLX  mean 
subscale  workload  were  excluded  only  because  of  their  very 
high  correlation  with  exact  match  errors  and  NASA  TLX, 
respectively. 

Examination  of  Table  11  shows  that  the  size  and 
statistical  significance  of  the  correlation  coefficients 


Table  11 

Correlation  Coefficients  Among  TAWL  Predictions  and 
Criterion  Measures 


TAWL  prediction 


Measure 

A 

Model 

VIS 

AUD 

KIN* 

COG 

PSY 

RMS 

6 

1 

-.450 

.450 

.872* 

.760 

.856* 

2 

-.450 

.450 

NR 

.778 

.919* 

EXERR 

4 

1 

.802 

NC 

.999* 

.992* 

.999* 

2 

.802 

NC 

NR 

.996* 

,995* 

TLX 

7 

1 

.367 

.109 

.921* 

.738 

.909* 

2 

.458 

.189 

NR 

.708 

.938* 

Note ■  The  following  are  used  as  headings:  VIS  >  Visual  ,  AUD  - 


Auditory  ,  KIN  ■  Kinesthetic,  COG  -  Cognitive  ,  PSY  -  Psychomotor  , 
AVE  -  Average,  RMS  •  Root  Mean  Squared  Tracking  Error,  EXERR  • 
Dichotic  Listening  (DL)  Exact  Match  Errors,  TLX  -  Task  Load  Index, 
and  NR  ■  Not  Rated.  The  Auditory  predictions  were  constant  for  DL 
trials;  Correlations  could  not  be  computed  (NC) . 

3 

Kinesthetic  ratings  were  not  assigned  in  TAWL  Model  2. 

*p<.05 


tend  to  be  consistent  across  the  two  models  and  the  three 
criterion  measures.  However,  the  size  of  the  correlation 
coefficients  vary  substantially  across  TAWL  component 
predictions . 

The  TAWL  visual  component  predictions  are  not 
significantly  related  to  any  of  the  three  measures.  This 
result  can  be  attributed  to  the  fact  that  the  model  only 
predicted  three  levels  of  visual  workload  for  all  the 
conditions.  The  lowest  level  was  predicted  for  the  Dxxx 
condition.  The  highest  level  was  predicted  for  the  three 
tracking  only  conditions  (xSxx,  xSPx,  xSPT) .  The 
intermediate  level  was  predicted  for  the  mixed  (DL  and 
tracking)  conditions  that  required  the  aviators  to  switch 
visual  attention  from  the  relatively  low  visual  workload  task 
of  finding  the  numbers  on  the  keypad  to  the  high  workload 
task  of  continuous  target  tracking.  The  switching  of  visual 
attention  from  the  tracking  task  to  the  numeric  entry  task 
reduces  visual  workload  and  results  in  greater  RMS  tracking 
error.  Thus,  the  negative  relationship  between  the  TAWL 
visual  component  predictions  and  RMS  tracking  error  is 
logically  consistent. 

The  TAWL  auditory  component  predictions  were  not  found 
to  be  significantly  correlated  with  either  of  the  two 
measures  for  which  correlation  coefficients  could  be 
computed.  As  was  suggested  for  the  visual  component 
predictions,  the  lack  of  a  statistically  significant 
relationship  between  the  measures  the  TAWL  auditory  component 
predictions  is  due  to  the  lack  of  variability  in  the  demands 
of  the  auditory  task  across  conditions.  That  is,  although 
overall  workload  varied  across  conditions,  the  model 
predicted  one  level  of  auditory  workload  when  the  condition 
required  the  dichotic  listening  task  and  a  second  level  when 
the  condition  did  not  require  the  performance  of  the  dichotic 
listening  task. 

As  has  been  found  in  previous  research,  when  kinesthetic 
and  psychomotor  ratings  are  used  in  TAWL  models,  the  workload 
predictions  generated  for  each  component  are  so  closely 
related  that  they  can  be  considered  redundant.  The 
similarity  of  the  two  component  predictions  is  indicated  by 
the  similarity  in  the  strength  of  their  relationships  with 
the  three  measures.  Table  11  shows  that  the  three  measures' 
correlation  with  the  TAWL  psychomotor  component  predictions 
is  nearly  identical  to  the  corresponding  measures' 
correlation  with  the  TAWL  kinesthetic  component  predictions/ 
the  correlation  coefficients  for  corresponding  measures 
differ  by  ±.02  or  less.  Nonetheless,  of  all  the  TAWL 
workload  components,  the  psychomotor  and  kinesthetic 


predictions  show  the  strongest  relationship  with  the 
criterion  measures. 

The  cognitive  predictions  were  most  closely  related  to 
the  DL  measure  and  were  not  found  to  be  correlated 
significantly  with  either  RMS  errors  or  TLX  workload  ratings. 
In  both  cases,  however,  the  correlation  coefficients  were 
nearly  large  enough  to  reach  statistical  significance. 

The  average  TAWL  predictions,  similar  to  the  psychomotor 
component  predictions,  show  a  consistently  strong 
relationship  with  each  of  the  criterion  measures.  The 
finding  that  predictions  for  the  average  TAWL  predictions  are 
related  to  the  criterion  measures  is  logically  consistent 
with  the  manner  in  which  task  demand  varied  across 
conditions . 

In  summary,  the  TAWL  workload  predictions  from  both 
models  show  strong  relationships  with  the  subjective,  and 
performance  workload  measures  used  as  criteria  for  TAWL 
validity.  Although  some  differences  between  models  produced 
by  different  analysts  were  apparent,  these  findings  strongly 
support  the  validity  of  the  TAWL  workload  prediction 
methodology. 


Discussion 

The  high  correlations  between  the  TAWL  model  predictions 
and  the  measures  of  workload  support  the  conclusion  that  the 
TAWL  methodology  generates  valid  predictions  of  operator 
workload.  However,  several  issues  raised  by  the  results  of 
the  research  are  addressed  before  stating  the  final 
conclusions.  The  discussion  is  organized  into  five 
subsections.  The  first  subsection  discusses  the  differences 
between  the  TAWL  models  produced  by  different  analysts.  The 
second  subsection  reviews  the  concept  of  overload  as  defined 
in  the  TAWL  methodology  in  light  of  the  aviator  performance. 
The  third  subsection  reviews  the  limitations  of  the  current 
research  design  and  the  fourth  subsection  makes 
recommendations  for  further  research.  The  final  subsection 
summarizes  the  major  conclusions  of  the  project. 


TAWL  Models  Constructed  by  Different  Analysts 

Differences  in  the  models  generated»by  different 
analysts  for  the  same  task  conditions  warrant  careful  review. 
If  a  methodology  does  not  constrain  the  analyst  sufficiently 
to  produce  valid  models,  the  method  itself  can  never  be 
validated,  only  the  individual  models.  However,  the 
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requirement  that  a  m.2vhodology  constrain  analysts 
sufficiently  to  produce  exactly  the  same  model  is  probably 
unreasonable.  Although  some  inter-analyst  differences  are  to 
be  expected,  the  differences  cannot  reduce  or  change  the 
nature  of  the  relationship  between  the  model's  predictions 
and  the  criterion  measures.  Described  below  are  three  types 
of  differences  between  the  models  that  were  identified. 

Structural  differences.  As  described  above,  the  TAWL 
methodology  uses  a  top-down  analysis  to  define  the  logical 
structure  of  the  actions  of  system  operators.  The  output  of 
the  analysis  is  a  list  of  the  tasks  that  operators  must 
perform.  In  the  process  of  identifying  the  individual  tasks 
in  a  system,  at  least  two  other  levels  of  analysis  are 
described  (e.g.,  segments,  functions). 

Defining  these  intermediate  levels  aids  analysts  in  at 
least  two  ways.  First,  the  levels  link  the  individual  tasks 
with  the  global  goals  of  the  system.  Second,  they  help  the 
analyst  manage  the  complexity  of  the  analysis.  Analyzing 
system  use  at  different  levels  greatly  reduces  demands  on  an 
analyst's  working  memory  while  allowing  for  the  management  of 
a  large  amount  of  complexity.  For  example,  it  is  a  simple 
matter  for  an  analyst  to  describe  the  sequencing  and 
interactions  of  the  5  to  15  functions  in  a  segment/  it  is  far 
more  difficult  to  describe  the  sequencing  and  interactions  of 
the  50  to  100  tasks  in  the  same  segment. 

Some  of  the  differences  between  the  models  generated  for 
this  research  were  differences  in  the  structure  of  the 
intermediate  levels.  Because  only  the  task  level  of  the 
analysis  is  directly  observable  as  operator  actions,  it  is 
difficult  to  verify  the  validity  of  higher  levels  of  the 
analysis.  In  other  words,  analysts  may  differ  in  the  manner 
in  which  they  group  tasks  into  functions;  however,  if  two 
different  functional  groupings  of  tasks  produce  the  same 
sequence  of  task  performance,  there  is  no  criterion  with 
which  to  determine  the  veracity  of  the  possible  candidates . 
Although  operator.*?  probably  use  logical  groupings  of  tasks 
that  might  be  determined  empirically,  the  exact  structure  of 
these  intermediate  levels  may  not  affect  the  predictions 
generated  by  the  model. 

The  analy-t  is  only  required  to  define  the  intermediate 
levels  in  a  way  that  generates  valid  task  sequencing  for  the 
operation  of  the  system.  Only  the  tasks  determine  TAWL 
workload  predictions;  analysts  do  not  assign  ratings  directly 
CO  either  the  functions  or  segments.  Thus,  if  one  analyst 
considered  a  set  of  16  tasks  to  be  one  function  and  another 
analyst  considered  the  tasks  to  be  two  functions,  the 
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validity  of  models  would  not  be  compromised  if  each  analyst 
used  the  functions  to  generate  equivalent  task  sequencing. 

This  discussion  highlights  the  importance  of  one  of  the 
procedures  recommended  by  the  TAWL  methodology.  The 
methodology  described  by  Hamilton,  Bierbaum,  and  Fulford 
(1991)  recommends  that  subject  matter  experts  (SMEs)  review 
and  approve  the  final  task  sequencing  generated  by  the 
simulation  of  segments.  In  conclusion,  differences  in  the 
structure  of  the  intermediate  levels  of  TAWL  models  may  be 
observed  without  necessarily  con^romising  the  validity  of  the 
models  so  long  as  they  produce  valid  task  sequences. 

Rating  differences.  The  second  type  of  difference 
between  the  models  were  rating  differences.  After  defining 
the  task  sequences,  analysts  use  workload  rating  scales  to 
rate  the  attentional  demands  of  each  task.  The  ratings  are 
made  for  components  that  describe  the  sensory,  cognitive,  and 
psychomotor  demands  of  tasks.  The  predictions  generated  by 
the  methodology  are  the  within-coroponent  sums  of  these 
ratings.  Thus,  differences  in  the  assigned  ratings  between 
analysts  directly  change  the  predictions  generated  by  the 
models . 

Rating  differences  between  analysts  appear  to  be  the 
most  serious  threat  to  the  validity  of  the  method.  However, 
the  differences  in  the  predictions  generated  by  the  models 
developed  for  this  research  did  not  substantially  reduce  the 
validity  of  either  model.  Apparently,  the  consistent  use  of 
the  rating  scales  by  both  analysts  was  sufficient  to  maintain 
the  validity  of  the  predictions.  This  is  surprising  because 
the  workload  predictions  generated  by  models  of  low 
complexity  are  more  highly  dependent  on  each  rating  than 
those  generated  by  models  of  high  complexity. 

A  relatively  small  number  of  ratings  formed  the  basis 
for  the  predictions  generated  by  the  models  used  here.  The 
workload  predictions  of  Model  1  and  Model  2  were  dependent  on 
a  total  of  35  and  24  ratings,  respectively.  Examples  of  the 
number  of  ratings  in  typical  TAWL  applications  range  from  828 
in  the  UH-60A  model  (Bierbaum,  Szabo,  &  Aldrich,  1989)  to 
3540  in  the  AH-64A  model  (Hamilton  &  Bierbaum,  1992) .  Large 
numbers  of  ratings  reduce  the  individual  impact  of  a  single 
rating , 

The  fact  that  rating  differences  did  not  seriously 
affect  the  validity  of  the  models  investigated  does  not  argue 
for  the  haphazard  use  of  the  rating  scales.  Indeed, 
consistent  use  of  the  scales  is  assumed  to  have  maintained 
the  validity  of  the  models.  Whenever  possible,  conclusions 
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should  not  be  drawn  from  comparisons  of  predictions  generated 
from  models  produced  by  different  analysts. 


Finally,  TAWL  methodology  recommends  that  the  workload 
ratings  be  made  by  the  consensus  of  more  than  one  analyst . 
This  procedure  improves  the  consistency  and  accuracy  of  the 
ratings  within  a  single  model.  Furthermore,  consensual 
rating  may  allow  for  the  comparison  of  TAWL  predictions 
generated  by  different  groups  of  raters  because  group  rating 
would  be  expected  to  reduce  individual  rating  bias. 

Opinion  differences.  Differences  of  opinion  were  the 
last  type  of  difference  between  the  models.  The  analysts 
constructed  different  models  to  reflect  their  opinions  about 
how  the  aviators  performed  the  tasks.  One  analyst  believed 
that  the  aviators  interrupted  the  tracking  tasks  while  they 
entered  the  DL  responses  on  the  keypad.  Because  DL  response 
entry  occupied  such  a  large  proportion  of  trial  time,  the 
other  analyst  assumed  that  task  sharing  would  have  to  occur 
and  he  did  not  Interrupt  the  tracking  tasks  during  DL 
response  entry. 

The  fact  that  this  difference  in  opinion  did  not  lead  to 
differential  validity  of  the  models  is  surprising.  The  total 
number  of  opinion  based  decisions  made  in  developing  the 
models  for  this  research  was  small;  each  had  the  potential  to 
change  substantially  the  predictions  generated  by  the  model. 
Again,  the  best  approach  to  minimizing  the  potential  effects 
of  differences  in  analyst  opinions  lies  in  the  consensual 
development  of  TAWL  models. 


The  Concent  of  Overload  in  the  TAWL  Methodnlnay 

The  concept  of  operator  overload  has  played  a  key  role 
in  the  TAWL  methodology  since  its  initial  development.  The 
notion  that  operators  can  maintain  good  performance  on  tasks 
until  they  reach  a  point  of  catastrophic  failure  was  popular 
in  Information  processing  and  workload  literature  when  TAWL 
was  originally  developed.  The  concept  is  intuitive,  is 
supported  by  laboratory  research,  and  is  useful  in 
Illustrating  the  detrimental  effects  of  workload  on 
performance . 

More  recently,  however,  the  workload  literature  has 
begun  to  recognize  that  good  performance  followed  by 
catastrophic  failure  is  only  one  of  many  different  ways  that 
operators  may  respond  when  presented  with  high  task  demands. 
Examination  of  the  entire  RMS  tracking  and  DL  performance 
functions  here  did  not  reveal  evidence  of  a  change  in 
performance  that  would  indicate  the  existence  or  level  of  an 
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overload  threshold,  even  though  both  models  predicted 
overloads  in  the  multi-task  conditions.  Clearly,  the  current 
findings  do  not  support  the  catastrophic  failure  concept  of 
workload. 

The  concept  of  operator  overload  was  useful  initially  in 
the  investigations  of  single  pilot  operation  of  scout/attack 
helicopters  (Aldrich  et  al.,  1984;  McCracken  &  Aldrich,  1984) 
because  of  rhe  unusually  high  demands  of  that  situation . 
However,  in  subsequent  investigations  of  production 
helicopters,  overload  conditions  were  rarely  observed  and 
other,  more  sensitive,  measures  of  workload  (e.g.,  average 
component  workload)  were  used  to  accomplish  the  objectives  of 
the  research. 

Synopsis .  Although  the  concept  of  task  overload  is 
useful  when  thinking  about  performance  on  individual  tasks, 
the  concept  of  a  fixed  level  of  task  demands  at  which 
performance  degrades  has  limited  utility  for  several  reasons. 
First,  its  genesis  was  in  the  analyses  of  a  very  high 
workload  situation  where  the  researchers  evaluated  different 
design  options.  There  are  other  measures  of  task  demands 
(e.g.,  mean  workload)  that  show  better  measurement  qualities 
(e.g.,  sensitivity)  for  a  broader  range  of  task  environments 
and  equipment  manipulations.  Second,  although  the  method 
used  the  overload  concept,  it  was  arbitrarily  placed  at  8  and 
it  has  never  been  empirically  determined.  Finally,  there  is 
growing  evidence  that  the  relationship  between  task 
performance  and  workload  is  complicated  and  cannot  be 
explained  using  any  single  concept.  As  such,  future 
applications  of  the  TAWL  methodology  should  reduce  reliance 
on  the  concept  of  operator  overload  and  continue  to  measure 
and  report  on  all  available  information  describing  the 
operator's  task  loading  and  performance  strategies. 


Limitations  of  the  Researrh  Design 

There  are  at  least  two  aspects  of  the  current  research 
design  that  limit  the  ability  to  generalize  the  results.  The 
first  and  most  fundamental  problem  was  the  artificial  task 
environment  used  in  the  research.  To  the  merit  of  the 
design,  the  tasks  were  selected  to  have  as  much  relevance  to 
the  operational  rotorcraft  environment  as  possible  and  to 
present  the  aviators  with  a  broad  range  of  workload. 
Unfortunately,  the  simplicity  of  the  environment  was  apparent 
in  the  lack  of  complexity  of  the  resulting  TAWL  models, 
especially  with  comparison  to  the  environments  in  which  TAWL 
has  typically  been  applied.  Until  a  TAWL  model  with  more 
typical  complexity  is  validated,  the  validity  of  complex  TAWL 
models  cannot  be  assumed. 
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Another  aspect  of  the  design  may  confound  the  findings 
of  the  research.  Because  access  to  the  aviators  was  limited 
and  because  the  aviators  were  unfamiliar  with  the 
experimental  tasks^  the  experimental  sessions  were  designed 
to  maximize  the  level  of  proficiency  obtained  in  the  time 
allotted.  Thus,  the  sequence  of  conditions  was  fixed  in 
every  experimental  session  and  was  arranged  in  a  manner 
designed  to  build  aviator  proficiency.  The  order  of 
conditions  was  determined  logically  by  the  author  and, 
judging  from  the  performance  data,  allowed  the  aviators  to 
reach  the  decelerating  part  of  the  learning  curve  before  the 
end  of  the  experiment . 

Unfortunately,  the  manipulation  of  workload  (through  the 
addition  of  tasks  to  conditions)  is  confounded  with 
presentation  order.  Thus,  other  factors  such  as  fatigue  or  a 
simple  order  effect  may  be  argued  to  account  for  the  variance 
in  the  data.  Randomization  of  the  conditions  on  the  final 
day  could  have  eliminated  the  confound  at  the  expense  of 
reducing  skill  acquisition.  Indeed,  some  instances  of 
catastrophic  failure  may  have  been  observed  if  aviators  had 
started  the  session  with  the  four-task  condition.  The  fact 
that  the  TAWL  predictions  and  most  of  the  measures  produce 
rank  orderings  of  the  conditions  that  differ  from  their 
presentation  order  supports  the  claim  that  the  intended 
manipulation  of  workload,  and  not  another  variable,  accounted 
for  the  observations  made  here. 


Suagestiona  foe.  Furthex-Reafiarch 

As  mentioned  above,  the  current  research  can  be 
interpreted  as  support  for  the  validity  of  the  TAWL 
methodology.  It  provides  a  valuable  first  step  in  the  full 
validation  of  the  methodology.  Of  course,  if  the  methodology 
had  not  produced  valid  models  in  this  task  environment,  then 
further  expense  for  validation  could  be  saved.  The  results, 
however,  justify  more  extensive  research. 

To  date,  two  research  plans  have  been  developed  that 
define  the  conduct  of  research  to  validate  TAWL  models. 
Aldrich  and  Szabo  (1986)  present  a  detailed  plan  for 
conducting  research  required  to  validate  the  one-crewmember 
light  helicopter,  experimental  (LHX)  workload  prediction 
model.  The  plan  explains  the  background,  defines  the 
problems,  and  presents  the  technical  objectives  to  be 
achieved  by  conducting  the  validation  research.  It  includes 
a  review  of  critical  issues  with  reference  to  the  workload 
literature.  The  conduct  of  the  research  was  planned  for  an 
advanced  research  simulator  and  includes  reference  to  174 


47 


objective  measures  of  performance  that  could  be  used  as 
criteria  for  validating  the  model. 

The  plan  calls  for  two  phases  of  simulator  research.  In 
the  first  phase,  segments  of  the  helicopter's  mission  would 
be  simulated  while  subjective  and  performance  measures  of 
workload  are  collected.  The  presentation  of  segments  would 
be  repeated  and  randomized.  In  the  second  phase,  full- 
mission  simulations  would  be  conducted  using  the  same 
measurement  protocol.  The  plan  calls  for  the  refinement  of 
the  model  whenever  sufficient  information  warrants. 

Hamilton  (1990)  developed  a  research  plan  to  test  the 
validity  of  the  AH-64A  workload  prediction  model.  He  modeled 
the  plan  after  that  of  Aldrich  and  Szabo's  (1986);  however, 
he  designed  it  to  be  conducted  in  the  AH-64A  combat  mission 
simulator  (CMS) .  Because  the  CMS  is  a  training  simulator 
rather  than  a  research  simulator,  very  few  objective  measures 
of  aircrew  performance  are  available  from  the  system. 
Therefore,  the  plan  describes  the  development  of  a 
performance  rating  scale  intended  to  be  used  as  a  criterion 
for  TAWL  validation.  The  scale  measures  the  tactical 
performance  of  the  crew  on  the  battlefield.  Hamilton  (199C) 
also  calls  for  two  phases  of  research  with  part-  and  full- 
mission  simulation. 

Recently,  the  U.S.  Army  Research  Institute  (ARl)  has 
acquired  a  unique  research  simulation  system.  The  simulator, 
previously  designated  the  simulator  complexity  testbed,  is 
now  referred  to  as  the  simulator  training  research  advanced 
testbed  for  aviation  (STRATA) .  The  testbed  simulates  an 
AH-64A  helicopter  and  was  designed  to  support  research  in 
training  and  simulator  design. 

Three  aspects  of  the  simulator  combine  to  make  it  the 
logical  site  for  further  TAWL  validation  research.  First, 
STRATA  simulates  an  AH-64A  helicopter  for  which  a  TAWL 
workload  prediction  model  and  validation  plan  already  exist. 

Second,  ARI  equipped  the  simulator  with  a  sophisticated 
data  recording  and  analysis  (DRA)  system  that  can 
automatically  capture  and  format  an  enormous  number  of 
measures  of  performance  (e.g.,  airspeed,  altitude,  ground 
track) .  The  DRA  can  also  measure  predefined  sequences  of 
cockpit  actions  of  interest.  For  example,  the  researcher  can 
program  the  DRA  to  report  the  average  time  between  enabling 
and  firing  the  weapon  systems  computed  over  the  entire 
mission  or  to  report  the  average  variation  in  hover  position 
measured  only  between  the  enabling  and  firing  of  the  missile 
system.  These  examples  are  not  suggested  as  measures  of 
performance.  They  demonstrate  that  the  DRA  system  offers 
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unique  possibilities  in  the  selection  of  aircrew  performance 
measures . 

The  final  aspect  of  STRATA  that  makes  it  an  excellent 
site  for  the  full  validation  of  the  TAWL  methodology  is  in 
the  richness  of  its  tactical  environment .  The 
characteristics  of  both  the  aircraft  and  tactical  environment 
influence  a  pilot's  workload.  The  tactical  environment  of 
most  simulators  quite  impoverished.  ARI  has  equipped  the 
STRATA  simulator  with  an  advanced  system  for  the  simulation 
of  the  tactical  environment  called  the  interactive  tactical 
environment  management  system  (ITEMS) .  The  system  controls 
the  tactical  environment  and  allows  for  the  control  of  the 
simulation  down  to  the  physics  of  individual  bullets.  ITEMS 
presents  the  aviators  with  unprecedented  tactical  realism 
through  the  use  of  artificially  intelligent  opponents  and 
allies.  This  intelligent  simulation  is  afforded  to  each  of 
the  defined  levels  of  organization  (e.g.,  crew,  platoon, 
company,  battalion,  etc.).  Besides  the  artificially 
intelligent  players  that  the  system  can  generate,  several 
auxiliary  stations  exist  that  allow  SMEs  to  "take  over"  any 
of  the  players  in  the  simulation  to  add  to  the  realism. 

In  summary,  two  well  defined  plans  exist  that  define 
explicit  conduct  of  TAWL  validation  research.  The  STRATA 
facility  is  recommended  as  the  site  for  the  research  because 
of  several  unique  advantages  of  the  system.  Finally,  the 
expense  required  to  update  the  plan  proposed  in  Hamilton 
(1990)  from  the  CMS  to  the  STRATA  should  be  minimal. 


Summary  and  Conclusions 

This  research  achieved  three  objectives.  First,  it 
tested  the  validity  of  the  TAWL  methodology  by  assessing  the 
relationships  between  subjective,  performance,  and 
physiological  measures  of  workload  and  the  predictions 
generated  by  two  TAWL  models.  Second,  it  examined  the 
differences  between  two  models  developed  by  different 
analysts  and  evaluated  the  implications  to  the  validity  of 
the  models.  Finally,  it  refined  and  extended  the  mechanisms 
used  in  the  methodology  to  link  workload  predictions  and 
performance.  The  conduct  of  the  research  resulted  in  the 
following  individual  findings: 

•  Seven  conditions  that  vary  in  workload  can  be 
constructed  from  combinations  of  four  tracking  and 
dichotic  listening  tasks. 

•  Two  qualified  analysts  were  able  to  use  the  TAWL 
methodology  to  construct  workload  prediction  models 
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and  generate  predictions  of  the  workload  in  each  of 
the  conditions. 

•  Three  types  of  differences  (structural,  rating,  and 
opinion)  were  found  in  the  construction  of  the  two 
models . 

•  The  models'  predictions  differed  but  were  highly 
correlated  (x  ■  .99). 

•  The  NASA  TLX  ratings  of  subjective  workload  were  well 
organized  and  sensitive  to  the  task  loading 
differences  between  the  conditions. 

•  The  RMS  measure  of  tracking  performance  consistently 
increased  as  a  function  of  condition  and  had  reached  a 
plateau  by  the  last  practice  session. 

•  Two  measures  of  dlchotic  listening  performance 
(omission  and  exact  match)  were  well  organized  and 
sensitive  to  task  condition. 

•  The  heart  rate  variability  measures  were  not  well 
organized  or  sensitive  to  task  condition. 

•  Three  measures  showed  sufficient  sensitivity  to  the 
across-condition  workload  manipulation  to  be  used  as 
criteria  for  TAWL  validity. 

•  The  correlations  among  the  3  criterion  measures  and 
the  average  of  TAWL  component  predictions  were  high 
(.89  <  X  <  .99)  for  both  models. 

These  findings  led  to  the  following  conclusions: 

•  The  TAWL  methodology  has  potential  for  generating 
valid  predictions  of  operator  workload. 

•  Although  differences  were  observed  between  the  models 
generated  by  different  analysts,  the  differences  did 
not  reduce  the  validity  of  either  model. 

•  Although  the  relationship  between  operator  workload 
and  performance  is  complex  (even  small  amounts  of 
workload  can  sometimes  degrade  performance),  the  TAWL 
methodolo^^y  provides  an  excellent  description  of  the 
aspects  of  the  task  environment  known  to  affect 
operator  performance. 
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The  validation  of  the  methodology  should  continue  with 
the  validation  of  a  TAWL  model  of  full  complexity  in 
the  STRATA  simulator  facility. 
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APPENDIX  A 


DEMOGRAPHIC  SURVEY 


A-1 


TAWL  WORKLOAD  PREDICTION  METHODOLOGY 
VALIDATION  RESEARCH 

AVIATOR  DEMOGRAPHIC  SURVEY 


INSTRUCTIONS:  This  survey  consists  of  questions  that  provide  information  about 
your  personal  background  and  experience.  Answer  each  Item  that  applies  to  you  by 
checking  in  the  appropriate  bracket  [  V  ]  or  by  printing  the  required  information  in  the 
space  provided.  When  answering  items  about  flight  hours,  you  may  refer  to  records,  if 
available,  or  you  may  estimate  the  flight  hours  as  ciosely  as  possible.  Your  responses 
will  be  used  for  research  purposes  only. 


1.  Name:  _ 

Last  First  Middle 

2.  Social  Security  Number: _ • _ - _ 

3.  Today's  Date:  _ 

Month  /  Day  /  Year 

4.  What  is  your  age  ? 

_ Years 


What  is  your  current  rank? 

[  ]  W01 

{  ]  2LT 

(  )  CW2 

I  ]  1LT 

(  ]  CW3 

I  ]  CPT 

[  ]  CW4 

[  ]  MAJ 

[  ]  LTC 

1  ]  COL 

To  which  unit  are  you  assigned? 

Unit: 

7.  How  long  have  you  been  assigned  to  your  present  Company? 

_ years  and _ months 

8.  Do  you  anticipate  reassignment  in  the  next  6  months? 

[  ]  Yes 

[  ]  NO 

If  yes,  give  expected  date  and  location  of  reassignment  _ 


9.  Currently,  what  is  your  primary  duty  position  in  the  unit? 


1 0.  What  additional  duties  do  you  perform  In  your  unit? 


1 1 .  How  long  have  you  been  on  active  duty  military  service? 

_ years  and _ months  of  active  service 

12.  How  long  has  it  been  since  you  graduated  from  initial  Army  flight  training? 

_ years  and _ months 

1 3.  How  long  has  it  been  since  you  graduated  from  the  AH'64  AQC? 

_ years  and _ months 

1 4.  Were  you  an  lERW  turnaround  student  in  the  AH-64  AQC? 

(  1  Yes 

[  ]  No 

If  no,  what  was  your  primary  aircraft  before  entering  the  Ah-64  AQC? 


1 5.  Indicate  the  total  number  of  flight  hours  you  have  logged  In  each  of  the  following 
aircraft.  Also,  check  [  V  ]  the  highest  duty  category  you  have  held  in  each 
aircraft. 


a.  Military  Rotary  Wing 

PI  PC  UT  IP  SI  IE 


AH-e4 

hours 

[ 

1 

( 

1 

1 

] 

( 

1 

[ 

] 

t 

] 

AH-I 

hours 

I 

1 

[ 

J 

I 

] 

I 

] 

[ 

) 

( 

1 

OH-58 

hours 

( 

] 

1 

J 

I 

] 

I 

] 

[ 

] 

1 

1 

UH-I 

hours 

I 

] 

{ 

J 

I 

] 

1 

J 

[ 

] 

[ 

] 

Other 

(Specify  other  aircra  ft; 

hours 

( 

] 

1 

] 

I 

] 

I 

1 

[ 

] 

[ 

1 

b.  Military  Fixed  Wing 

U-21  hours  [  ]  (  J  I  J  [  1  [  ]  I  ] 

C-12  -  hours  1  i  I  J  I  )  [  1  (  I  I  1 

OV-1  _  hours  [  1  (  1  (  )  [  1  t  1  1  ) 

Other  -  hours  [1  (I  |  )  (  )  (  )  (  ] 

(Speclly  other  aircraft) _ 


1 6.  How  many  flight  hours  have  you  logged  in  each  seat  of  the  AH-64? 

Front  Seal _ hours 

Backseat _ hours 

1 7.  How  many  flight  hours  have  you  logged  in  each  seat  of  the  AH-64  CMo? 

Front  Seat _ hours 

Back  Seat _ hours 

1 6.  How  many  flight  hours  have  you  iogged  in  each  seat  of  the  AH-64  CWEPT? 

Front  Seat _ hours 

Back  Seat _ hours 

1 9.  What  is  your  current  crew  station  designation? 

[  ]  AH-64  front  seat 

I  ]  AH*64  back  seat 

I  ]  Both  seats  (expiain) _ 

20.  What  is  your  current  Readiness  Level? 

(  ]  RL1 

I  1  RL2 
I  )  RL3 

21 .  Have  you  been  assigned  to  a  fixed  crewmate? 

I  ]  Yes 

[  ]  No 

22.  If  you  are  a  member  of  a  fixed  crew,  how  many  hours  has  your  crew  trained 
together? 

_ flight  hours 

_ CMS  hours 
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Workload  Prediction  Validation  Research 


NASA-Taek  Load  Index 
Rating  Scales 


Orientation:  During  this  research,  we  are  interested  in  assessing  your  performance 
and  your  experiences  during  individual  segments  of  a  tracking  and  listening  test  battery. 
Below,  the  technique  that  will  be  used  to  examine  your  experiences  is  described.  The 
technique  was  developed  by  NASA  and  is  called  the  Task  Load  Index  or  TLX  for  short. 
In  the  most  general  sense,  we  want  to  examine  your  Vorkload”  experience.  Workload 
is  a  difficult  concept  to  define  precisely,  but  a  simple  one  to  understand.  The  factors 
that  influence  your  experience  of  workload  may  come  from  the  task  Itself,  your  feelings 
about  your  own  performance,  how  much  effort  you  put  in,  or  the  stress  and  frustration 
you  feel.  The  workload  contributed  by  different  tasks  may  change  as  you  get  more 
familiar  with  a  task,  perform  easier  or  harder  versions  of  the  task,  or  move  from  one  task 
to  another.  Physical  workload  is  relatively  easy  to  conceptualize  and  evaluate. 

However,  mental  workload  is  more  subjective,  making  it  more  difficult  to  measure. 

Because  workload  Is  something  that  is  an  individual  experience,  effective  “rulers" 
of  workload  are  difficult  to  define.  Previous  research  has  identified  at  least  six  factors 
that  may  contribute  to  workload.  Because  workload  may  be  caused  by  any  of  these 
factors,  we  would  like  you  to  evaluate  each  of  them  rather  than  lumping  them  into  an 
overall  evaluation  of  workload.  The  set  of  six  rating  scales  described  on  the  following 
page  was  developed  to  evaluate  your  experiences  during  different  task  segments. 
Please  read  the  descriptions  of  the  scales  carefully.  If  you  have  a  question  about  any  of 
the  scales  in  the  table,  please  ask  me  about  It.  It  is  extremely  Important  that  they  be 
clear  to  you.  You  may  keep  the  descriptions  with  you  for  reference  during  the 
experiment. 


Instructions:  For  each  task  segment  that  you  perform,  you  will  be  given  a  sheet  of  six 
rating  scales  similar  to  the  one  pictured  above.  After  each  segment,  you  will  evaluate 
the  segment  by  putting  a  mark  on  each  of  the  six  scales  at  the  point  which  matches 
your  experience.  Each  scale  has  two  single  word  descriptors  that  anchor  each 
endpoint.  Please  consider  your  responses  carefully  in  distinguishing  among  the 
different  task  segments.  Consider  each  scale  individually.  Your  ratings  will  play  an 
important  role  in  the  evaluation  being  conducted.  Thus,  your  active  participation  is 
essential  to  the  success  of  this  experiment  and  is  greatly  appreciated. 
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NASA-Task  Load  Index 
Rating  Scales 


Rating  Scale  Definitions 

Title 

Endpoints 

Descriptions 

MENTAL 

DEMAND 

Low /High 

How  much  mental  and  perceptual  activity  was 
required  (e.g.,  thinking,  deciding,  calculating, 
remembering,  looking,  searching,  etc.)?  Was 
the  task  easy  or  demanding,  simple  or  complex, 
forgiving  or  exacting? 

PHYSICAL 

DEMAND 

Low /High 

How  much  physical  activity  was  required  (e.g., 
pushing,  pulling,  turning,  controlling,  activating, 
etc.)?  Was  the  task  easy  or  demanding,  slow  or 
brisk,  slad(  or  strenuous,  restful  or  laborious? 

TEMPORAL 

DEMAND 

Low /High 

How  much  time  pressure  did  you  feel  due  to  the 
rate  or  pace  at  which  the  tasks  or  task  elements 
occurred?  Was  the  pace  slow  and  leisurely  or 
rapid  and  frantic? 

PERFORMANCE 

Poor /Good 

How  successful  do  you  think  you  were  in 
accomplishing  the  goals  of  the  task  set  by 
yourself?  How  satisfied  were  you  with  your 
performance  in  accomplishing  these  goals? 

EFFORT 

Low /High 

How  hard  did  you  have  to  work  (mentally  or 
physically)  to  accomplish  your  level  of 
performance? 

FRUSTRATION 

LEVEL 

Low /High 

How  secure,  gratified,  content,  relaxed  and 
complacent  versus  insecure,  discouraged, 
irritated,  stressed  and  annoyed  did  you  feel 
during  the  task? 
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NASA-Task  Load  Index 
Sources  of  Workload 


Orientation:  Throughout  this  research,  six  rating  scales  were  used  to  assess  your 
experiences  in  the  different  task  segments.  Scales  of  this  sort  are  extremely  useful,  but 
their  utility  suffers  from  the  tendency  people  have  to  interpret  them  in  individual  ways. 
For  example,  some  people  feel  that  mental  or  temporal  demands  are  the  essential 
aspects  of  workload  regardless  of  the  effort  they  expended  on  a  given  task  or  the  level 
of  performance  they  achieved.  Others  feel  that  if  they  performed  well,  the  workload 
must  have  been  low  and  if  they  performed  badly,  it  must  have  been  high.  Yet  others 
feel  that  effort  or  feelings  of  frustration  are  the  most  important  factors  in  workload,  and 
so  on.  In  addition,  the  factors  that  create  levels  of  workload  differ  depending  on  the 
task.  For  example,  some  tasks  might  be  difficult  because  they  must  be  completed  very 
quickly.  Others  may  seam  easy  or  hard  because  of  the  intensity  of  mental  or  physical 
effort  required.  Yet  others  feel  difficult  because  they  cannot  be  performed  well,  no 
matter  how  much  effort  is  expended. 

Instructions:  The  evaluation  you  are  about  to  perform  is  a  technique  that  has  been 
developed  by  NASA  to  assess  the  relative  importance  of  the  six  factors  in  determining 
your  overall  workload.  The  procedure  is  simple:  You  will  be  presented  with  a  series  of 
pairs  of  rating  scale  titles  (for  example.  Effort  or  Mental  Demand)  and  asked  to  choose 
which  of  the  items  was  more  important  to  your  overall  experience  of  workload  in  the 
segments  that  you  performed.  Each  pair  of  scale  titles  will  appear  on  a  separate  card. 

Circle  the  scale  title  that  represents  the  more  important  contributor  to  workload 
for  the  segments  you  performed  in  this  research. 

After  you  have  finished  the  entire  series,  we  will  be  able  to  use  the  pattern  of  your 
choices  to  create  a  weighted  combination  of  your  segment  ratings  to  generate  an  overail 
workload  score.  Please  consider  your  choices  carefuliy  and  make  them  consistent  with 
how  you  used  the  rating  scales  during  the  task  segments  you  were  asked  to  evaluate. 
Don’t  think  that  there  is  any  correct  pattern;  we  are  only  interested  in  your  opinions. 

if  you  have  any  questions,  please  ask  them  now.  Otherwise,  start  whenever  you 
are  ready.  Thank  you  for  your  participation. 


Task  1.  -  Stick  Only 


MENTAL  DEMAND 


Low  High 


PHYSICAL  DEMAND 


Low 


TEMPORAL  DEMAND 


. I  I  I  I  I  I  I  I  I  I  I  I 


u 


Low 


High 


PERFORMANCE 


Poor 


I  JJ  l-l  J  J.L1J  I 


Good 


EFFORT 


Low  High 


FRUSTRATION 


APPENDIX  C 


EXPERIMENTAL  CONTROL  FILE 


Appendix  C  contains  the  computer  file  that  controlled 
the  experimental  sessions  of  the  TAWL  workload  validation 
research.  The  file  contains  two  types  of  controlling 
instructions  that  are  placed  in  the  first  column  of  the  file. 
The  I  instructions  indicate  that  the  following  material  is  an 
instruction  screen.  The  E  instructions  Indicate  that  the 
next  line  in  the  file  contains  the  specification  for  the  next 
experimental  trial.  Experimental  conditions  were  created  by 
selecting  the  dichotic  listening  (D) ,  stick  (S) ,  pedals  (P), 
or  throttle  (T)  tasks  to  be  included  in  the  trail  using  the 
Dn  Sn  Pn  Tn  command.  Replacing  the  n  with  either  a  1  or  a  0 
would  activate  or  deactivate  the  taskS/  respectively. 


c-i 


During  this  uptriaant  you  will  bo  porfonoing  7  dtfforont  tooko.  You 
will  bo  ropooting  ooch  took  3  tiaoo  for  o  tetol  of  21  triolo.  Each  triol 
tokoo  3  ainutoo.  Tho  oxporiatnt  io  ontiroty  oolf-pocod.  You  dotonoino 
uhon  you  porfoni  tho  noxt  triol.  You  aay  toko  to  1  ainuto  botuoon 
triolo  and  finloh  in  tho  allotted  tiao.  Ao  an  aid  in  kooping  paco,  on 
olapood  tiao  clock  la  provldod  on  ooch  ocroon. 

Uo  oro  recording  your  heart  rate,  roapirationo.  and  eye  blinko,  ploooo 
notify  the  oxporlaantor  if  tho  adhooivo  on  any  of  tho  oloctrodoo  faila. 

Ploooo  check  that  the  RED  wire  ontoro  tho  hoadoat  on  tho  right. 

Roaoabar  your  concentration  and  ioprovaaont  on  thaoo  taaka  ia 
inportant  to  tho  eutcona  of  thla  axporiaant.  Try  your  beat. 

Uhon  you  ora  ready  for  tho  Took  1  inotruetiona...Praoo  tho  i1  key 


In  thio  axporiaant  only  tho  cantor  controller  (whore  tho  cyclic  in  a 
hollcoptor  would  bo)  will  bo  active.  Obtain  a  coafortabla  oaating 
pooitlon  placing  your  foot  on  tho  padalo  and  lot  your  loft  hand  root  to 
your  aide.  The  trigger  and  buttono  on  tho  controller  have  no  affect  on 
tho  coaputor.  Oraop  tho  control lor  handle  with  your  right  hand  placing 
your  index  finger  on  tho  trigger.  Uhon  tho  trial  bogino,  uoo  tho  handle 
to  control  tho  aovoaonta  of  tho  eroao  in  tho  central  portion  of  tho 
coaputor  ocroon.  Try  to  aolntain  tho  crooo  in  tho  cantor  of  tho  ocroon 
in  line  with  tho  crooo  haira. 

Tho  coaputor  hoo  a  built  in  drift  coaponant  oo  you  are  following  a 
aoving  target. 

Uhon  you  are  ready  to  begin  Took  1  •  Triol  1 
Prooa  the  #1  keyl 
E 

DO  SI  PO  TO 
I 


Took  1  ■  Triol  1  eoaplotodl 


Relax,  taka  a  brook  for  a  ainuto... 


Uhon  you  are  ready  to  begin  Took  1  -  Trial  2 
Proto  tho  f1  koyl 


E 

00  Si  PO  TO 
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Task  1  -  Trial  2  coaptatadi 


Ralax,  taka  a  braak  for  a  alnuta... 


Whan  you  ara  raady  to  bogln  Taak  1  •  Trial  S 
Praaa  tha  i1  kayl 


E 

DO  SI  PO  TO 
I 


Taak  1  •  Trial  3  coaplatadl 


*  Taka  a  ainuta  to  eoaplata  tha  NASA'TLX  ratlnp  acala  for  * 

•  Taak  1  * 


Whan  you  ara  raady  for  tha  Taak  2  Inttructiona... 
Praaa  tha  #1  kay 


I 

j  *  Taak  3  -  liataning  Only 


In  this  axpariaant  you  Mill  ba  liitaninp  to  aata  of  lattara  and  diglta 
spokan  ovar  tha  haadphonaa  and  antaring  your  raaponaaa  on  tha  nuiter 
kaypad  uith  your  loft  hand.  During  aaeh  aat,  lattara  and  diglta  ara 
praaantad  to  aach  oar  alultanaoualy.  TOUR  IS  TO  fOCUt  YOUR 

ATTENTION  ON  ONE  EAR  AND  TO  ENTER  ONLY  THE  DIOITS  HEARD  THROUON  THAT  EAR 
ON  THE  KEYPAD.  Tha  aar  you  focua  on  changaa  froa  aat  to  tat  and  will  ba 
claarly  idantifiad  at  RIGHT  or  LEFT  bafora  aaeh  aat  of  lattara  and  diglta 
bagina.  Uaa  your  indax  fingar  to  racord  tha  diglta  diglta  you  hoar  by 
pratting  tha  kay  corratponding  to  tha  digit.  Praaa  tha  appropriata  kay 
aa  toon  aa  yo>i  haar  tha  digit. 

Raantter,  tha  lattar  'O'  it  HOT  tha  nj*ar  'ZERO' 

Whan  you  ara  raady  to  bogln  Taak  2  *  Trial  1 

Praaa  tha  f1  kayl 


E 

01  to  PO  TO 
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I 


TMk  2  -  Trfal  1  ca^>l«tadl 


t*k*  •  bTMk  for  a  alnuta... 


Uhar  you  art  rtt&f  to  bagin  Task  2  *  Trial  2 
froaa  tho  it  Koyi 
E 

01  SO  PO  TO 
I 


Task  2  •  Trial  2  eoaplatadi 


Ralax,  taka  a  brtak  for  a  ainuta... 


Whan  you  ara  raady  to  bagin  Task  2  *  Trial  S 
Proas  tha  ft  kayt 
E 

01  SO  PO  TO 


Task  2  -  Trial  3  eoaplatadl 


*  Taka  a  ainuta  to  coaplata  tha  NAM'TLX  rating  acala  for  * 

•  Taak  2  • 


Whan  you  ara  raady  for  tha  Taak  3  tnatructiona... 
Prasa  tha  #1  kay 
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I 


■ 


TMk  3  *  LUttning  *  Stick 


You  will  now  porfoni  Task  1  (otick)  and  Taak  2  (iiataning)  in  dual. 
Try  to  do  aach  ta^  equally  wall.  You  wmy  axparianca  difficulty,  but  do 
your  vary  boat.  First  tha  stick  task  will  Inittata.  A  faw  saconds  latar 
latar,  tha  llstanlng  task  will  start.  Tha  Matanlng  task  will  and  Just 
bafora  tha  stick  task.  Sa  cartain  to  continua  parfortslns  tha  stick  task 
inti  I  tha  vidso  display  disappoars  fras  tha  scraan. 

Whsn  you  ara  ready  to  begin  Task  3  ■  Trial  1 
Press  tha  #1  kayl 


E 

01  Si  PO  TO 
I 


Taak  3  *  Trial  1  coaptatadl 


Relax,  taka  a  break  for  a  ainuta. .. 


Whan  you  are  ready  to  begin  Task  3  *  Trial  2 
Press  tha  f1  kayl 


E 

01  Si  PO  TO 
I 


Task  3  -  Trial  2  coa^latadl 


Ralax,  taka  a  break  for  a  ■Inuta... 


When  you  are  ready  to  begin  Task  3  ■  Trial  3 
Press  the  f1  kayl 


E 

01  SI  PO  TO 
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J 


TMk  3  •  Trial  3  eoaplatadi 


*  Taka  a  alnuta  to  cooplata  tha  HAtA-TLX  rating  acala  for  * 

*  Taak  3  • 


Uhan  you  ara  raady  for  tha  Taak  4  Inatructiona... 
Praaa  tha  #1  kay 


Taak  4  -  Stick  «  hadala 


In  thia  axparlaant  only  tha  cantar  controllar  and  tha  padala  Mill  ba 
activa.  Obtain  a  coafortabla  taatlng  position  placing  your  right  hand  on 
tha  cantar  controllar.  Placa  your  faat  on  tha  tMO  foot  padala,  so  that 
your  faat  raat  halfsay  on  tha  padala  and  your  haala  on  tha  floor.  Thaao 
foot  padala  ulll  control  tha  dlractlon  a^  apaad  of  tha  foot  padal 
:uraor.  Praaoing  on  tha  laft  padal  anvaa  tha  cursor  right.  Prasaing  on 
tha  right  padal  aovas  tha  cursor  laft.  Uhan  tha  trial  baglna,  you  ulll 
parfona  tha  stick  tracking  task  as  bafora.  You  mIU  also  saa  tha  foot 
pedal  cursor  nova  across  tha  bottosi  of  tha  scraan.  You  asjot  kaap  tha 
foot  padal  cursor  as  cloaa  as  possible  to  tha  vartical  bar  In  tha  cantar 
of  tha  scraan.  Tha  coisputar  has  s  built  in  drift  eogpenant  so  you  ara 
following  taovlng  targats. 

Whan  you  ara  raady  to  begin  Task  4  «  Trial  1 
PrsM  tha  f1  kayl 
E 

DO  SI  PI  TO 

1 


Task  4  -  Trial  I  coaplatadl 


Ralax,  taka  a  break  for  a  alnuta 


Uhan  you  ara  raady  to  begin  Task  4  •  Trial  2 

Press  tha  kayl 
E 

00  t1  PI  TO 
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I 


Tank  4  •  Trial  2  Maplatad! 


Ralaa.  taka  a  braak  for  a  alnuta... 


Whan  you  ara  raady  to  boflln  Took  4  -  Trial  3 
Rraaa  tha  #1  kayi 


E 

00  $1  PI  TO 
I 


Task  4  *  Trial  3  eoRplatadl 


*  Take  a  alnute  to  eoifpleta  tha  NAM'TLX  ratios  aoala  for  * 

*  Took  4  • 


Uhon  you  ara  raady  for  tha  Taak  5  Inatructlena... 
Praaa  tha  f1  kay 


I 


Task  5  •  Listenino  *  Stick  *  Padata 


In  thia  ojiparlaant  you  Mill  bo  parfenalng  tha  llatanins  taak,  tha 
•tick  taak,  and  rna  padal  taak  In  eotd>lnation.  Try  to  do  aach  taak 
aqually  wall. 

Vilen  you  ara  raady  to  basin  Taak  S  ■  Trial  1 
Praaa  tha  f1  kayf 


E 

D1  81  PI  TO 
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Uh«n  you  or*  roody  to  bofln  Took  S  ■  Trial  2 
ProM  tiM  i1  ktyl 
E 

01  SI  PI  TO 
I 


Took  S  *  Trial  2  cai|>latadl 


Whan  you  aro  ready  to  baftn  Task  5  '  Trial  3 
Praaa  tlia  f1  kayt 
E 

01  $1  PI  TO 
I 


Task  5  •  Trial  3  coaiplatodf 


Whan  you  ara  ready  for  tha  Task  6  inatruetiona... 
Praaa  tha  f1  kay 
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I 


Task  6  -  Stick  *  PcdaU  ♦  Threttla 


In  thU  cxpcriMnt  th«  canter  contreUar,  tha  padala,  and  tha  right 
ana  ecntrellar  hUI  ba  active.  You  Mill  parfor*  tha  stick  tracking  and 
foot  pedal  taaka  at  bafora.  Poaitien  your  right  hand  and  fact  at  In  the 
prtvioua  tattt.  Whan  tha  trial  bagina,  you  ulll  atao  aaa  tha  throttle 
cursor  nova  along  tha  left  edge  of  tha  scraan.  Tha  throttle  is 
controlled  by  tha  handle  located  to  your  left.  Taka  hold  of  tha  throttle 
by  closing  your  LEFT  hsnd  around  the  handle  ao  that  tha  handle  is  insida 
your  left  fist  and  your  index  finger  is  on  tha  trigger.  Pressing  tha 
stick  forward  aovas  tha  cursor  ip;  pulling  it  toward  you  aovas  it  down. 
You  asist  keep  tha  throttle  cursor  as  close  as  poasibla  to  tha  horitontal 
bar  In  tha  cantor  of  the  scraan. 

Whan  you  are  ready  to  begin  Task  6  •  Trial  1 
Prase  tha  f1  kayl 


E 

DO  SI  PI  T1 
I 


Task  6  •  Trial  1  coaplatadl 


[: 


Relax,  taka  a  break  for  a  ainuta. . . 


Whan  you  are  ready  to  begin  Task  6  *  Trial  2 
Press  tha  f 1  kayl 


E 

00  II  PI  T1 
I 


Task  6  *  Trial  2  coaplatadl 


Relax,  taka  a  break  for  a  ainuta... 


Whan  you  era  ready  to  begin  Task  6  *  Trial  3 
Press  tha  #1  kayl 


E 

00  SI  PI  T1 
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I 


Tuk  6  -  Trial  9  coaptatadl 


*  Taka  a  ainuta  to  ceaplata  tha  NAtA'TLX  rating  acala  for  * 

•  Taak  6  * 


WKc:<  you  ara  raady  for  tha  Taak  7  Inatructlena... 
Praaa  tha  #1  kay 


*  Taak  7  •  Llataning  *  Stick  *  Padala  a  Throttla  * 


Itc  all  on  noMi  In  this  axparlaant  tha  cantar  controllar,  laft 
control lor,  and  tha  pedals  Mill  activa.  •.  addition  tha  llataning  task 
Mill  ba  activa.  It  Is  alanst  lopoaslbla  'o  control  tha  throttle  and 
antar  nudiars  on  tha  keypad  at  tha  asaia  tlaa  unlaas  you  uso  your  foraans 
or  uaa  soaw  othar  Innovation.  This  It  a  difficult  taak.  Try  to  do  your 
bast. 

Whan  you  ara  raady  to  bagln  Taak  7  •  Trial  1 
Prats  tha  01  kayl 

£ 

01  SI  PI  T1 

1 


Task  7  *  Trial  1  coiplatadl 


Ralax,  taka  a  braak  for  a  olnuta... 


Uhan  you  ara  raady  to  bagln  Task  7  •  Trial  Z 
Praaa  tha  01  kayl 


E 

01  Si  PI  T1 
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TMk  7  •  Trial  2  coaplatadi 


Rtlax,  taka  a  braak  for  a  ■inuta... 


Whan  you  ara  raatfy  to  baflin  Taak  7  *  Trial  S 
Proat  tha  01  kayl 
E 

01  SI  Pi  T1 
I 


Taak  7  •  Irlal  S  coa^latadl 


*  Taka  a  ailnuta  to  eoaplata  tha  NASA'TU  ratlns  aeala  for  * 

*  Taak  7  • 


Whan  you  havt  coiqplatad  tha  NASA'TLX  rating... 
Praaa  tha  01  kay 
I 


*  Thank  you  for  your  participation  In  tha  * 

•  • 

*  Taak  Analyala  /  Workload  Prediction  Validation  Roaaarch  * 


That'a  all  folkal 
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APPENDIX  D 
TAWL  Model  1 
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Segment  Decision  Rules  D-10 
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Preliminary  TAWL  Validation  Workload  Prediction  Model  -  CRB 

TASK  DATABASE 


Task 

COG 

PSY 

VIS 

AUD 

KIN 

1. 

CONTROL  STICK 

1.20 

2.60 

0.00 

0.00 

1.00 

2. 

CONTROL  THROTTLE 

1.20 

2.60 

0.00 

0.00 

1.00 

3. 

RECEIVE  RANDOM  NUMBERS 

5.30 

0.00 

0.00 

4.90 

0.00 

4  . 

MONITOR  AUDIO 

1.00 

0.00 

0.00 

1.00 

0.00 

5. 

ENTER  RANDOM  NUMBERS 

1.20 

2.20 

3.70 

0.00 

1.00 

6. 

CONTROL  PEDALS 

1.20 

2.60 

0.00 

0.00 

1.00 

7. 

MONITOR  CRT 

1.20 

0.00 

5.40 

0.00 

0.00 
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Preliminary  TAWL  Validation  Workload  Prediction  Model  *-  CRB 
Function  Decision  Rules 


Function  1:  CONTROI.  VERTICAL  MOVEMENT 
Discrete  Fixed  Tasks 

Entry  #  Task  #  Task  Name  Cretnnember  Start 

1  2  CONTROL  THROTTLE  PILOT  (P)  0.0 


Duration 

-0.5 


Preli^ninary  TAWt  Validation  Workload  Prediction  Model  -  CRB 
Function  Decision  Rules 


Discrete 
Entry  # 
1 


Function  2:  CONTROL  HORIZONTAL  MOVEMENT 


Fixed  Tasks 


Task  # 

Task  Namo 

Crewmember 

Start 

6 

CONTROL  PEDALS 

PILOT  (P) 

o 

o 
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Duration 

-0.5 


Preliminary  TAWL  Validation  Workload  Prediction  Model  -  CRB 
Function  Decision  Rules 


Function  3:  CONTROL  RETICLE  MOVEMENT 
Discrete  Fixed  Tasks 

Entry  #  Task  <  Task  Name  Crewmember  Start 

1  1  CONTROL  STICK  PILOT  (P)  0.0 


Duration 

-0.5 


Prollminary  TANL  Validation  Workload  Prediction  Model  -  CRB 
Function  Decision  Rules 


Function  4:  ENTERING  NUMBERS 

Discrete  Fixed  Tasks 

Entry  #  Task  #  Task  Name  Crewmember  Start 

1  S  ENTER  RANDOM  NUMBERS  PILOT  (P)  0.0 


Duration 

1.0 


Preliminary  TAWL  Validation  Workload  Prediction  Model  -  CRB 
Function  Decision  Rules 


Function  S:  MOMll'OR  VISUAL 

Discrete  Fixed  Tasks 

Entry  #  Task  #  Task  Name  Crewmember  Start 

I  7  MONITOR  CRT  PILOT  (P)  0.0 


Duration 

-0.5 
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Preliminary  TAHL  Validation  Workload  Prediction  Model  -  CRB 
Function  Decision  Rules 


Function  6:  MONITOR  AUDIO 

Discrete  Fixed  Tasks 

Entry  #  Task  #  Task  Name  Crewmember  Start 

1  4  MONITOR  AUDIO  PILOT  (P)  0.0 


Duration 

-0.5 
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Prelininary  TAHL  Validation  NorAload  Pradiction  Model  •  CRB 
Function  Decision  Rules 


Function  7:  RECEIVE  NUMBERS 


Discrete  Fixed  Tasks 


Entry  # 

Task  « 

Task  Name 

Crewneieber 

Start 

Duration 

1 

3 

RECEIVE  RANDOM 

NUMBERS 

PILOT  (P) 

0.0 

12.5 

2 

4 

MONITOR  AUDIO 

PILOT  (P) 

12.5 

1.5 

3 

3 

RECEIVE  RANDOM 

NUMBERS 

PILOT  (P) 

14.0 

4.0 

4 

4 

MONITOR  AUDIO 

PILOT  (P) 

18.0 

2.5 
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Preliminary  TANL  Validation  Workload  Prediction  Model  '  CRB 

Segment  Decision  Rules 


Segment  1:  OL  ONLY 


Discrete  Fixed  Functions 

#  Function  Name 

Start 

Duration 

Interrupted  By 

€  MONITOR  AUDIO 

0.0 

8.0 

7 

7  RECEIVE  NUMBERS 

8.0 

20.5 

7  RECEIVE  NUMBERS 

28.5 

20.5 

7  RECEIVE  NUMBERS 

49.0 

20.5 

7  RECEIVE  NUMBERS 

89.5 

20.5 

7  RECEIVE  NUMBERS 

90.0 

20.5 

7  receive  NUMBERS 

110.5 

20.5 

7  RECEIVE  NUMBERS 

131.0 

20.5 

7  RECEIVE  NUMBERS 

x51.5 

20.5 

6  MONITOR  AUDIO 

172.0 

-8.0 

Discrete  Random  Functions 

#  Function  Name 

Duration 

Start 

Finish 

Times  Interrupted  By 

4  ENTERING  NUMBERS 

1.5 

10.0 

20.0 

5 

4  ENTERING  NUMBERS 

1.5 

21.0 

28.5 

4 

4  ENTERING  NUMBERS 

1.5 

30.5 

40.5 

5 

4  ENTERING  NUMBERS 

1.4 

41.5 

49.0 

4 

4  ENTERING  NUMBERS 

1.5 

51.0 

81.0 

5 

4  ENTERING  NUMBERS 

1.5 

82.0 

69.5 

4 

4  ENTERING  NUMBERS 

1.5 

71.5 

81.5 

5 

4  ENTERING  NUMBERS 

1.5 

82.5 

90.0 

4 

4  ENTERING  NUMBERS 

1.5 

92.0 

102.0 

5 

4  ENTERING  NUMBERS 

1.5 

103.0 

110.5 

4 

4  ENTERING  NUMBERS 

1.5 

112.5 

122.5 

5 

4  ENTERING  NUMBERS 

1.5 

123.5 

131.0 

4 

4  ENTERING  NUMBERS 

1.5 

133.0 

143.0 

5 

4  ENTERING  NUMBERS 

1.5 

144.0 

151.5 

4 

4  ENTERING  NUMBERS 

1.5 

153,5 

163.5 

5 

4  ENTERING  NUMBERS 

1.5 

164.5 

180.0 

4 
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Preliminary  TAHL  Validation  tforkload  Prediction  Model  >  CRB 

Segment  Decision  Rules 


Segment  2:  STICK  ONLY 
Continuous  Fixed  Functions 

#  Function  Name  Start  Duration  Interrupted  By 

3  CONTROL  RETICLE  MOVEMENT  0.0  180.0 

5  MONITOR  VISUAL  0.0  180.0 
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Preliminary  TANL  Validation  Workload  Prediction  Model  ->  CRB 

Segment  Decision  Rules 

Segment  3;  DL  AND  STICK 


Discrete  Fixed  Functions 

#  Function  Name 

Start 

Duration 

Interrupted  By 

6  MONITOR  AUDIO 

0.0 

8.0 

7 

7  RECEIVE  NUMBERS 

S.O 

20.5 

7  RECEIVE  NUMBERS 

28.5 

20.5 

7  RECEIVE  NUMBERS 

48.0 

20.5 

7  RECEIVE  NUMBERS 

69.5 

20.5 

7  RECEIVE  NUMBERS 

90.0 

20.5 

7  RECEIVE  NUMBERS 

110.5 

20.5 

7  RECEIVE  NUMBERS 

131.0 

20.5 

7  RECEIVE  NUMBERS 

151.5 

20.5 

€  MONITOR  AUDIO 

172.0 

-8.0 

Continuous  Fixed  Functions 

#  Function  Name 

Start 

Duration 

Interrupted  By 

3  CONTROL  RETICLE  MOVEMENT 

0.0 

-180.0 

5  MONITOR  VISUAL 

0.0 

180.0 

4 

Discrete  Random  Functions 

#  Function  Name 

Duration 

Start 

Finish 

Times  Interrupted  By 

4  E^'1'ERING  NUMBERS 

1.0 

10.0 

20.0 

5 

4  ENTERING  NUMBERS 

1.0 

21.0 

28.5 

4 

4  ENTERING  NUMBERS 

1.0 

30.5 

40.5 

5 

4  ENTERING  NUMBERS 

1,0 

41.5 

49.0 

4 

4  ENTERING  NUMBERS 

1.0 

51.0 

61.0 

5 

4  ENTERING  NUMBERS 

1.0 

62.0 

69.5 

4 

4  ENTERING  NUMBERS 

1.0 

71,5 

81.5 

5 

4  ENTERING  NUMBERS 

1.0 

82.5 

90.0 

4 

4  ENTERING  NUMBERS 

1.0 

92.0 

102.0 

5 

4  ENTERING  NUMBERS 

1.0 

103.0 

110.5 

4 

4  ENTERING  NUMBERS 

1.0 

112.5 

122.5 

5 

4  ENTERING  NUMBERS 

1.0 

123.5 

131.0 

4 

4  ENTERING  NUMBERS 

1.0 

133.0 

143.0 

5 

4  ENTERING  NUMBERS 

1.0 

144.0 

151.5 

4 

4  ENTERING  NUMBERS 

1.0 

153.5 

163.5 

5 

4  ENTERING  NUMBERS 

1.0 

164.5 

180.0 

4 

D-12 


Preliminary  TAWL  Validation  Norhload  Prediction  Model  -  CRB 

Segment  Decision  Rules 


Segment  4:  STICK  AND  PEDAL 


Continuous  Fixed  Functions 


* 

Function  Name 

Start 

Duration 

interrupted  By 

1 

CONTROL  VERTICAL  MOVEMENT 

0.0 

180.0 

3 

CONTROL  RETICLE  MOVEMENT 

0.0 

180.0 

5 

MONITOR  VISUAL 

0.0 

180.0 

D-13 


Pr«Iiitiinary  TANL  Validation  Norkload  Prediction  Model  -  CRB 

Segment  Decision  Rules 


Segment  5:  DL,  STICK  AND  PEDAL 


Discrete  Fixed  Functions 

#  Function  Name 

Start 

Duration 

Interrupted  By 

6  MONITOR  AUDIO 

0.0 

8.0 

7 

7  RECEIVE  NUMBERS 

e.o 

20.5 

7  RECEIVE  NUMBERS 

2S.5 

20.5 

7  RECEIVE  NUMBERS 

49.0 

20.5 

7  RECEIVE  NUMBERS 

69.5 

20.5 

7  RECEIVE  NUMBERS 

90.0 

20.5 

7  RECEIVE  NUMBERS 

110.5 

20.5 

7  RECEIVE  NUMBERS 

131.0 

20.5 

7  RECEIVE  NUMBERS 

151.5 

20.5 

6  MONITOR  AUDIO 

172.0 

8.0 

Continuous  Fixed  Functions 


« 

Function  Name 

Start 

Durrtion 

Interrupted  By 

1 

CONTROL  VERTICAL 

MOVEMENT 

0.0 

180.0 

4 

3 

CONTROL  RETICLE  MOVEMENT 

0.0 

-180.0 

5 

MONITOR  VISUAL 

0.0 

180.0 

4 

Discrete  Random  Functions 

* 

Function  Name 

Duration 

Start 

Finish 

Times  Interrupted  By 

4 

ENTERING  NUMBERS 

1.0 

10.0 

20.0 

5 

4 

ENTERING  NUMBERS 

1.0 

21.0 

28.5 

4 

4 

ENTERING  NUMBERS 

1.0 

30.5 

40.5 

5 

4 

ENTERING  NUMBERS 

1.0 

41.5 

49.0 

4 

4 

ENTERING  NUMBERS 

l.O 

51.0 

61.0 

5 

4 

ENTERING  NUMBERS 

i.O 

62.0 

69.5 

4 

4 

ENTERING  NUMBERS 

1.0 

71.5 

81.5 

5 

4 

ENTERING  NUMBERS 

1.0 

82.5 

90.0 

4 

4 

ENTERING  NUMBERS 

1.0 

92.0 

102.0 

5 

4 

ENTERING  NUMBERS 

1.0 

103.0 

110.5 

4 

4 

ENTERING  NUMBERS 

1.0 

112.5 

122.5 

5 

4 

ENTERING  NUMBERS 

1.0 

123.5 

131.0 

4 

4 

ENTERING  NUMBERS 

1.0 

133.0 

143.0 

5 

4 

ENTERING  NUMBERS 

1.0 

144.0 

151.5 

4 

4 

ENTERING  NUMBERS 

1.0 

153.5 

163.5 

5 

4 

ENTERING  NUMBERS 

1.0 

164.5 

180.0 

4 

0-14 


Ptsliminacy  TAWL  Validation  Norkload  Pradiction  Modal  -  CRB 

Sagment  Oaciaion  Rulas 

Sagmant  €:  STICK,  PEDAL  AND  THROTTLE 
Continuous  Fixed  Functions 

#  Function  Naim  Start  Duration  Inter ruptad  By 

1  CONTROL  VERTICAL  MOVEMENT  0.0  180.0 

2  CONTROL  HORIZONTAL  MOVEMENT  0.0  180.0 

3  CONTROL  RETICLE  MOVEMENT  0.0  180.0 

5  MONITOR  VISUAL  0.0  180.0 
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Prelinin«ry  TAlOi  Validation  Workload  Prediction  Model  -  CRB 

Segment  Decision  Rules 


Segment  7:  DL,  STICK,  PEDAL  AND  THROTTLE 


Discrete  Fixed  Functions 

#  Function  Name 

Start 

Duration 

Interrupted  By 

6  MONITOR  AUDIO 

0.0 

8.0 

7 

7  RECEIVE  NUMBERS 

8.0 

20.5 

7  RECEIVE  NUMBERS 

28.5 

20.5 

7  RECEIVE  NUMBERS 

49.0 

20.5 

7  RECEIVE  NUMBERS 

69.5 

20.5 

7  RECEIVE  NUMBERS 

90.0 

20.5 

7  RECEIVE  NUMBERS 

110.5 

20.5 

7  RECEIVE  NUMBERS 

131.0 

20.5 

7  RECEIVE  NUMBERS 

151.5 

20.5 

6  MONITOR  AUDIO 

172.0 

-8.0 

Continuous  Fixed  Functions 

#  Function  Name 

Start 

Duration 

Interrupted  By 

1  CONTROL  VERTICAL 

MOVEMENT 

0.0 

180.0 

4 

2  CONTROL  HORIZONTAL  MOVEMENT 

0.0 

180.0 

3  CONTROL  RETICLE  MOVEMENT 

0.0 

-180.0 

5  MONITOR  VISUAL 

0.0 

180.0 

4 

Discrete  Random  Functions 

#  Function  Name 

Duration 

Start 

Finish 

Times  Interrupted  By 

4  ENTERING  NUMBERS 

1.0 

10.0 

20.0 

5 

4  ENTERING  NUMBERS 

1.0 

21.0 

28.5 

4 

4  ENTERING  NUMBERS 

1.0 

30.5 

40.5 

5 

4  ENTERING  NUMBERS 

1.0 

41.5 

49.0 

4 

4  ENTERING  NUMBERS 

1.0 

51.0 

61.0 

5 

4  ENTERING  NUMBERS 

1.0 

62.0 

69.5 

4 

4  ENTERING  NUMBERS 

1.0 

71.5 

81.5 

5 

4  ENTERING  NUMBERS 

1.0 

82.5 

90.0 

4 

4  ENTERING  NUMBERS 

1.0 

92.0 

102.0 

5 

4  ENTERING  NUMBERS 

1.0 

103.0 

110.5 

4 

4  ENTERING  NUMBBPS 

1.0 

112.5 

122.5 

5 

4  ENTERING  NUMBERS 

1.0 

123.5 

131.0 

4 

4  ENTERING  NUMBERS 

1.0 

133.0 

143.0 

5 

4  ENTERING  NUMBERS 

1.0 

144.0 

151.5 

4 

4  ENTERING  NUMBERS 

1.0 

153.5 

163.5 

5 

4  ENTERING  NUMBERS 

1.0 

164.5 

180.0 

4 
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APPENDIX  E 
TAWL  Model  2 


DATA _ Page 

Task  Database  E-  2 
Function  Decision  Rules  E-  3 
Segment  Decision  Rules  E-11 


Preliminary  TAWL  Validation  Workload  Prediction  Model  -  DBH 

TASK  DATABASE 


Task 

Aud 

Via 

Cog 

1.  Control  Vertical  Position 

0.00 

0.00 

1.00 

2.  Control  Horizontal  Position 

0.00 

0.00 

1.00 

3.  Monitor  Audio 

1.00 

0.00 

0.00 

4 .  Detect  Numbers 

4.90 

0.00 

4.60 

5.  Enter  Numbers 

0.00 

5.00 

1.20 

6 .  Monitor  Screen 

0.00 

5.40 

0.00 

E-2 


Pay 


2.60 

2.60 

0.00 

0.00 

2.20 

0.00 


Preliminary  TANL  Validation  Korkload  Prediction  Model  -  DBH 
Function  Decision  Rules 

Function  1 :  Stick 


Continuous  Random  Tasks 

Start  :  0.0 

Finish:  -0.5 

Crewmember:  AVIATOR  (A) 

Entry  #  Task  #  Task  Name  Duration 

1  1  Control  Vertical  Position  O.S 

2  2  Control  Horizontal  Position  0.5 


E-3 


Prellminery  TAWL  Validation  Norkload  Pcadletion  Model  -  DBH 


Continuous  Fixed 
Entry  #  Task  # 
1  2 


Function  Decision  Rules 

Function  2:  Pedals 


Tasks 


Task  Nans 

Crewineinber 

Start 

Duration 

Control  Horizontal  Position 

AVIATOR 

0.0 

-0.5 

E-4 


Preliminary  TANL  Validation  Workload  Prediction  Model  -  DBH 
Function  oeciaion  Rulea 


Function  3:  Throttle 


Continuous 

Fixed 

Tasks 

- 

Entry  « 

Task  # 

Task  Name 

Crewmember 

Start 

Duration 

1 

1 

Control  Vertical  Position 

AVIATOR  (A) 

0.0 

-0.5 

E-5 


-'S’*. 


Preliminary  TANL  Validation  Workload  Prediction  Model  -  DBH 
Function  Decision  Rules 

Function  4:  Detect  Numbers 


Discrete  Fixed  Tasks 


Entry  # 

Task  # 

Task  Nam 

CreMBtember 

Start 

Duration 

1 

3 

Monitor  Audio 

AVIATOR  (A) 

0.0 

1.5 

2 

4 

Detect  Numbers 

AVIATOR  (A) 

1.5 

11.0 

3 

3 

Monitor  Audio 

AVIATOR  (A) 

12.5 

1.5 

4 

4 

Detect  Numbers 

AVIATOR  (A) 

14.0 

4.0 
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Preliminary  TANL  Validation  Workload  Prediction  Model  -  DBH 
Function  Decialon  Rules 


Function  5: 

Monitor  Audio 

Continuous  Fixed 

Tasks 

Entry  #  Task  i 

Task  Haste 

CretRoesiber 

Start 

Duration 

1  3 

Monitor  Audio 

AVIATOR  (A) 

0.0 

-0.5 

E-7 


Preliminary  TAWL  Validation  Workload  Prediction  Model  *>  DBH 
Function  Deciaion  Rules 


Function  6: 

Monitor  Screen 

Continuous 

Fixed 

Tasks 

Entry  # 

Task  1 

Task  Name 

Crewmember 

Start 

Duration 

1 

6 

Monitor  Screen 

AVIATOR  (A) 

0.0 

-0.5 

E-8 


Pr«llninary  TAffL  Validation  Norkload  Prediction  Model  •  DBH 
Function  Decision  Rules 

Function  7:  Enter  Number 

Discrete  Fixed  Tasks 


Entry  # 

Task  # 

Task  Name 

CrsMBember 

Start 

Duration 

1 

5 

Enter  Numbers 

AVIATOR  (A) 

0.0 

0.5 

Prallminary  TAWL  Validation  Workload  Pradietion  Nodal  -  DBH 
Function  Daciaion  Rulaa 


Function  8:  Entar  Nunbars 


Dlacrata  Flxad  Tasks 


Entry  # 

Task  # 

Task  Nama 

Crawmanajar 

Start 

Duration 

1 

5 

Entar  Nunbars 

AVIATOR  (A) 

0.0 

1.0 

2 

5 

Entar  Nunbars 

AVIATOR  (A) 

1.0 

1.0 

3 

5 

Entar  Nunbars 

AVIATOR  (A) 

2.0 

1.0 

4 

5 

Entar  Numbers 

AVIATOR  (A) 

3.0 

1.0 

E-i: 


Pralimlnary  TANL  Validation  Workload  Pradiction  Model  -  DBH 
Segment  Decision  Rules 


Segment  1:  Stick  (ST) 


Continuous  Fixed  Functions 
#  Function  Name 
1  Stick 

€  Monitor  Screen 


Start  Duration  Interrupted  By 

0.0  -0.5 

0.0  -180.0 


E-11 


Preliminary  TANL  Validation  Workload  Prediction  Model  -  DBH 

Segment  Decision  Rules 


Segment  2 : 

Diehotic 

listening 

(DL) 

Discrete  Fixed  Functions 

#  Function  Name 

Start 

Duration 

Interrupted  By 

4  Detect  Numbers 

8.0 

18.0 

8  Enter  Numbers 

24.5 

4.0 

4  Detect  Numbers 

28.5 

18.0 

8  Enter  Numbers 

45.0 

4.0 

4  Detect  Numbers 

49.0 

18.0 

8  Enter  Numbers 

65.5 

4.0 

4  Detect  Numbers 

69.5 

18.0 

8  Enter  Numbers 

86.0 

4.0 

4  Detect  Numbers 

90.0 

18.0 

8  Enter  Numbers 

106.5 

4.0 

4  Detect  Numbers 

110.5 

18.0 

8  Enter  Numbers 

127.0 

4.0 

4  Detect  Numbers 

131.0 

18.0 

8  Enter  Numbers 

147.5 

4.0 

4  Detect  Numbers 

151. 5 

18.0 

8  Enter  Numbers 

168.0 

4.0 

Continuous  Fixed  Functions 

#  Function  Name 

Start 

Duration 

Interrupted  By 

5  Monitor  Audio 

0.0 

-180.0 

4 

Discrete  Random  Functions 

#  Function  Name  Duration 

Start 

Finish 

Times  interrupted  By 

7  Enter  Number  0.5 

11.5 

22.5 

5 

7  Enter  Number  0 . S 

32.0 

43.0 

5 

7  Enter  Number  0.5 

52.5 

63.5 

5 

7  Enter  Number  0 . 5 

73.0 

84.0 

5 

7  Enter  Number  0 . 5 

93.5 

104.5 

5 

7  Enter  Number  0.5 

114.0 

125.0 

5 

7  Enter  Number  0 . 5 

134.5 

145.5 

5 

7  Enter  Number  0.5 

155.0 

166.0 

5 
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Preliminary  TAWL  Validation  Norkload  Prediction  Model  -  OBH 

Segment  Decialon  Rules 


Segment  3:  ST  and  DI> 


Discrete  Fixed  Functions 

#  Function  Name 

Start 

Duration 

Interrupted  By 

4  Detect  Numbers 

S.O 

18.0 

8  Enter  Numbers 

24.5 

4.0 

4  Detect  Numbers 

28.5 

18.0 

8  Enter  Numbers 

45.0 

4.0 

4  Detect  Numbers 

49.0 

18.0 

8  Enter  Numbers 

65.5 

4.0 

4  Detect  Numbers 

69.5 

18.0 

8  Enter  Numbers 

86.0 

4.0 

4  Detect  Numbers 

90.0 

18.0 

8  Enter  Numbers 

106.5 

4.0 

4  Detect  Numbers 

110.5 

18.0 

8  Enter  Numbers 

127.0 

4.0 

4  Detect  Numbers 

131.0 

18.0 

8  Enter  Numbers 

147.5 

4.0 

4  Detect  Numbers 

151.5 

18.0 

8  Enter  Numbers 

168.0 

4.0 

Continuous  Fixed  Functions 

#  Function  Name 

Start 

Duration 

Interrupted  By 

1  Stick 

0.0 

•0.5 

6  Monitor  Screen 

0.0 

-180.0 

7  8 

5  Monitor  Audio 

0.0 

-0.5 

4 

Discrete  Random  Functions 

#  Function  Name  Duration 

Start 

Finish 

Times  Interrupted  By 

7  Enter  Number  0.5 

11.5 

22.5 

5 

7  Enter  Number  0.5 

32.0 

43.0 

5 

7  Enter  Number  0.5 

52.5 

63.5 

5 

7  Enter  Number  0.5 

73.0 

84.0 

5 

7  Enter  Number  0 . 5 

93.5 

104.5 

5 

7  Enter  Number  0.5 

114.0 

125.0 

5 

7  Enter  Number  0.5 

134.5 

145.5 

5 

7  Enter  Number  0 . 5 

155.0 

166.0 

5 
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Pr«liinlnary  TAKL  Validation  Workload  Prediction  Model  -  DBH 

Segment  Decision  Rules 

Segment  4:  ST  and  Pedals  (PD) 


Continuous  Fixed  Functions 


#  Function  Name 

Start 

Duration  Interrupted  By 

1  Stick 

0.0 

-0.5 

2  Pedals 

0.0 

-0.5 

6  Monitor  Screen 

0.0 

-180.0 
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Preliminary  TANL  Validation  Workload  Prediction  Model  -  dbh 
Segmant  Decision  Rules 


Segment 

5:  ST, 

OL,  and  PD 

Discrete  Fixed  Functions 

*  Function  Name 

Start 

Duration 

Interrupted  By 

4  Detect  Numbers 

8.0 

18.0 

8  Enter  Numbers 

24.5 

4.0 

4  Detect  Numbers 

28.5 

18.0 

8  Enter  Numbers 

45.0 

4.0 

4  Detect  Numbers 

49.0 

18.0 

8  Enter  Numbers 

65.5 

4.0 

4  Detect  Numbers 

69.5 

18.0 

8  Enter  Numbers 

86.0 

4.0 

4  Detect  Numbers 

90.0 

18.0 

8  Enter  Numbers 

106.5 

4.0 

4  Detect  Numbers 

110.5 

18.0 

8  Enter  Numbers 

127.0 

4.0 

4  Detect  Numbers 

131.0 

18.0 

8  Enter  Numbers 

147.5 

4.0 

4  Detect  Numbers 

151.5 

18.0 

8  Enter  Numbers 

168.0 

4.0 

Continuous  Fixed  Functions 


I  Function  Name 

1  Stick 

2  Pedals 

6  Monitor  Screen 

5  Monitor  Audio 

Start 

0.0 

0.0 

0.0 

0.0 

Duration 

-0.5 

-0.5 

-180.0 

-0.5 

Interrupted  By 

7  8 

4 

Discrete  Random  Functions 

4  Function  Name 

Duration 

Start 

Finish 

Times 

Interrupted  By 

7  Enter  Number 

0.5 

11.5 

22.5 

5 

7  Enter  Number 

0.5 

32.0 

43.0 

5 

7  Enter  Number 

0.5 

52.5 

63,5 

5 

7  Enter  Number 

0.5 

73.0 

84.0 

5 

7  Enter  Number 

0.5 

93.5 

104.5 

5 

7  Enter  Number 

0.5 

114.0 

125,0 

5 

7  Enter  Number 

0.5 

134.5 

145,5 

5 

7  Enter  Number 

0.5 

155.0 

166.0 

5 
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Preliminary  TAWt  Validation  Workload  Prediction  Model  -  OBH 

Segment  Decision  Rules 

Segment  6:  ST,  PO,  and  Throttle  (TR) 


Continuous  Fixed  Functions 


Function  Name 

Start 

Duration 

Stick 

0.0 

>0.5 

Pedals 

0.0 

-0.5 

Throttle 

0.0 

-0.5 

Monitor  Screen 

0.0 

-180.0 

Interrupted  By 
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Preliminary  TANL  Validation  Workload  Prediction  Model  -  DBH 

Segment  Dec ia ion  Rules 


Segment  7: 

ST,  DL, 

PD,  and  TR 

Discrete  Fixed  Functions 

#  Function  Name 

Start 

Duration  interrupted  By 

4  Detect  Numbers 

8.0 

18.0 

8  Enter  Numbers 

24.5 

4.0 

4  Detect  Numbers 

28.5 

18.0 

8  Enter  Numbers 

45.0 

4.0 

H  Detect  Numbers 

49.0 

18.0 

8  Enter  Numbers 

65.5 

4.0 

4  Detect  Numbers 

69.5 

18.0 

8  Enter  Numbers 

86.0 

4.0 

4  Detect  Numbers 

90.0 

18.0 

8  Enter  Numbers 

106.5 

4.0 

4  Detect  Numbers 

110.5 

18.0 

8  Enter  Numbers 

127.0 

4.0 

4  Detect  Numbers 

131.0 

18.0 

8  Enter  Numbers 

147.5 

4.0 

4  Detect  Numbers 

151.5 

18.0 

8  Enter  Numbers 

168,0 

4.0 

Continuous  Fixed  Functions 


#  Function  Name 

1  Stick 

2  Pedals 

3  Throttle 

6  Monitor  Screen 

5  Monitor  Audio 

Start 

0.0 

0.0 

0.0 

0.0 

0.0 

Duration 

-0.5 

-0.5 

-0.5 

-180.0 

-0.5 

interrupted  By 

7  8 

4 

Discrete  Random  Functions 

*  Function  Name 

Duration 

Start 

Finish 

Times 

Interrupted  By 

7  Enter  Number 

0.5 

11.5 

22.5 

5 

7  Enter  Number 

0.5 

32.0 

43.0 

5 

7  Enter  Number 

0.5 

52.5 

63.5 

5 

7  Enter  Number 

0.5 

73,0 

84.0 

5 

7  Enter  Number 

0.5 

93.5 

104.5 

5 

7  Enter  Number 

0.5 

114.0 

125.0 

5 

7  Enter  Number 

0.5 

134.5 

145.5 

5 

7  Enter  Number 

0.5 

155.0 

166.0 

5 
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APPENDIX  F 


CORRELATION  MATRIX 

Appendix  F  contains  the  correlation  matrix  for  37  variables 
measured  during  the  TAWL  preliminary  velidation  research.  The 
intersection  of  each  row  and  column  of  the  matrix  contains  three 
numbers.  The  first  number  is  the  Pearson  product-moment 
correlation  coefficient  (r)  of  the  row  and  column  variables;  the 
second  number  is  the  probability  (ft)  of  the  correlation;  the 
third  number  is  the  number  (ft)  of  pairs  used  to  calculate  the 
correlation. 


F-l 


Root  Meem 
Squared 
Error 


Root  Mean 
Sq[uared 
Error 

Omission 

Error 


Commission 

Error 


Exact  I 

Error 

Mental 

Demand 


Physical 

Demand 


Tenporal 

Demand 


Own 

Performance 


Omission 

Error 


0.99649 

0.0533 

3 


Effort 


Frustration! 


Subscale 

Workload 

Time-Based 

IBI 


IBI 

Standard 

Deviation 

IBI 
Coeff . 
Variance 


0.99649 

0.0533 

3 

0.78871 

0.4215 

3 

0.99569 

0.0592 

3 

0.97373 

0.0010 

6 

0.97751 

0.0006 

6 

0.97880 

0.0007 

6 

-0.85684 

0.0293 

6 

0.97128 

0.0012 

6 

0.97606 

0.0009 

6 

0.98965 

0.0002 

6 

0.98537 

0.0003 

6 

-0.70355 

0.1188 

6 

0.08364 

0.8748 

6 

0.47264 

0.3438 

6 


Commis¬ 

sion 

Error 


0.78871 

0.4215 

3 

0.84551 

0.1545 

4 


0.84551 

0.1545 

4 

0.98140 

0.0186 

4 

0.99815 

0.0019 

4 

0.99338 

0.0066 

4 

0.99791 

0.0021 

4 

-0.95961 

0.0404 

4 

0.99122 

0.0088 

4 

0.99966 

0.0003 

4 

0.99913 

0.0009 

4 

0.99880 

0.0012 

4 

-0.74479 

0.2552 

4 

-0.69428 

0,3057 

4 

-0,52429 

0.4757 

4 


Exact 
Match 
Error 


0.99569 

0.0592 

3 

0.98140 

0.0186 

4 

0.91203 

0.0880 

4 


0.91203 

0.0880 

4 

0.82055 

0.1795 

4 

0.89222 

0.1078 

4 

0.87797 

0.1220 

4 

-0.86142 

0,1386 

4 

0.90705 

0.0930 

4 

0.85735 

0.1426 

4 

0.86658 

0.1334 

4 

0.87058 

0,1294 

4 

-0.89126 

0.1087 

4 

-0.86405 

0.1360 

4 

-0.68243 

0.3176 

4 


0.977673 


Mental  Physical 
Demand  Demand 


0.97373  0.97751 

0.0010  0.0008 


0.99815  0.99338 

0.0019  0.0066 


0.82055  0.89222 

0.1795  0.1078 


0.97767  0.99687 

0.0223  0.0031 


0.96984 


0.0223 

4l 

0,0003 

7 

0.99687 

0.0031 

4 

0.96984i 

0.00031 

7: 

Pi'",” : 

0.98835 

0.0116 

4 

0.99032 
'  0,0001 
7 

0.95873 

0.0006 

7 

-0.98659 

0.0134 

4 

-0.92079 

0.0032 

7 

-0.86974 

0.0110 

7 

0.98942 

0.0106 

4 

0.97031 

0.0003 

7 

0.99360 

0.0001 

7 

0.98602 

0.0140 

4 

0.99503 

0.0001 

7 

0.98361 

0.0001 

7 

0.98785 

0.0322 

4 

0.98377 

O.OOOi 

7 

0.98850 

0.0001 

7 

0.98818 

0.0118 

4 

0.99082 

0.0001 

7 

0.99183 

0.0001 

7 

-0.75446 

0.2455 

4 

-0.68553 

0.0891 

7 

-0.74225 

0,0561 

7 

-0.81843 

0.1816 

4 

-0.31001 

0.4986 

7 

-0.44021 

0.3229 

7 

-0.66650 

0.3335 

4 

-0,02970 

0.9496 

7 

-0.14103 

0.7630 

7 

/  s  '  '■''  ' 

.  '«'  -T\’’ 

Root  Mean 
Squared 
Error 

Omission 

Error 

Coamis- 

sion 

Error 

Exact 

Match 

Error 

Mental 

Demand 

Physical 

Demand 

RMS 

0.90783 

0.19788 

-0.16771 

0.01127 

0.49572 

.34723 

Succ.  Inter. 

0.0124 

0.8021 

0.8323 

0.9887 

0.2579 

0.4454 

Differences 

6 

4 

4 

4 

7 

7 

RMS 

0.94123 

0.43161 

0.14741 

0.26863 

0.66583 

0.54566 

Coeff . 

0.0051 

0.5684 

0.8526 

0.7314 

0.1025 

0.2052 

Variance 

6 

4 

4 

4 

7 

7 

Sun 

0.92865 

0.52786 

0.11004 

0.35575 

0.69103 

0.58597 

Succ.  Inter 

0.0075 

0.4721 

0.8900 

0.6443 

0.0856 

0.1668 

Differences 

6 

4 

4 

4 

7 

7 

Decel .  / 

0.74315 

0.31610 

-0.17469 

0.12961 

0.43370 

0.30798 

Fluctuations 

0.0905 

0.6839 

0.8253 

0.8704 

0.3310 

0.5016 

6 

4 

4 

4 

7 

7 

Frequency- 

-0.91684 

-0.96081 

-0.49050 

-0.92826 

-0.86002 

-0.83613 

Based 

0.0101 

0.0392 

0.5095 

0.0717 

0.0130 

0.0191 

IBI 

6 

4 

4 

4 

7 

7 

Respiration 

-0.72368 

-0.40604 

-0.99551 

-0.22660 

-0.62117 

-0.74963 

Band 

0.1040 

0.5940 

0.0045 

0.7734 

0.1365 

0.0524 

6 

4 

4 

4 

7 

7 

Blood 

0.30607 

0.04035 

-0.92270 

0.19462 

0.06022 

-0.19370 

Pressure 

0.5552 

0.9596 

0.0773 

0.8054 

0.8979 

0.6773 

Band 

6 

4 

4 

4 

7 

7 

Model  1 

0.76029 

0.94972 

0.93752 

0.99216 

0.80940 

0.68945 

Cognitive 

0.0793 

0.0503 

0.0625 

0.0078 

0.0274 

0.0866 

6 

4 

4 

4 

7 

7 

Model  1 

0.85620 

0.98337 

0.90784 

0.99993 

0.85851 

0.94847 

Psychomotor 

0.0295 

0.0166 

0.0922 

0.0001 

0.0134 

0.0011 

6 

4 

4 

4 

7 

7 

Model  1 

-0.44989 

0.67257 

0.89232 

0.80206 

0.25576 

0.46989 

Visual 

0.3707 

0.3274 

0.1077 

0.1979 

0.5799 

0.2874 

6 

4 

4 

4 

7 

7 

Model  1 

0.44989 

• 

0.30742 

0.10142 

Auditory 

0.3707 

6 

4 

4 

4 

0.5024 

7 

0.8287 

7 

Model  1 

0.87234 

0.98337 

0.90784 

0.99993 

0.87410 

0.95766 

Kinesthetic 

0.0234 

0.0166 

0.0922 

0.0001 

0.0101 

0.0007 

6 

4 

4 

4 

7 

7 

Model  1 

0.90495 

0.93697 

0.94279 

0.98662 

0.95146 

0.93965 

Mean 

0.0131 

0.0630 

0.0572 

0.0134 

0.0010 

0.0017 

Workload 

6 

4 

4 

4 

7 

7 

Model  1 

0.85581 

0.97689 

0.78305 

0.92365 

0.61812 

0.74946 

Overload 

0.0297 

0.0231 

0.2169 

0.0763 

0.0245 

0.0524 

Conditions 

6 

4 

4 

4 

7 

7 

Model  1 

0.85102 

0.97356 

0.80227 

0.92365 

0.81263 

0.74570 

Overload 

0.0316 

0.0264 

0.1977 

0.0764 

0.0263 

0.0543 

Density 

6 

4 

4 

4 

7 

7 

»  -  R 

Root  Mean 
Squared 
Error 

Omission 

Error 

Commis¬ 

sion 

Error 

Exact 

Match 

Error 

Mental 

Demand 

Physical 

Demand 

Model  2 

0.77810 

0.98520 

0.92383 

0.99548 

0.78626 

0.64913 

Cognitive 

0.0684 

0.0148 

0.0762 

0.0045 

0.0360 

0.1147 

€ 

4 

4 

4 

7 

7 

Model  2 

0.91949 

0.98520 

0.92383 

0.99548 

0.89018 

0.97165 

Psychomotor 

0.0095 

0.0148 

0.0762 

0.0045 

0.0072 

0.0003 

6 

4 

4 

4 

7 

7 

Model  2 

-0.44989 

0.67257 

0.89232 

0.80206 

0.36306 

0.55187 

Visual 

0.3707 

C .3274 

0.1077 

0.1979 

0.4234 

0.1990 

6 

4 

4 

4 

7 

7 

Model  2 

0.44989 

0.30742 

0.10142 

Auditory 

0.3707 

6 

4 

4 

4 

0.5024 

7 

0.8287 

7 

Model  2 

0.88511 

0.93865 

0.96050 

0.98604 

0.93182 

0.90516 

Mean 

0.0190 

0.0614 

0.0395 

0.0140 

0.0022 

0.0051 

Workload 

6 

4 

4 

4 

7 

7 

Model  2 

0.79158 

0.86791 

0.47444 

0.78883 

0.77000 

0.69132 

Overload 

0.0606 

0.1321 

0.5256 

0.2112 

0.0429 

0.0854 

Conditions 

6 

4 

4 

4 

7 

7 

Model  2 

0.79158 

0.86791 

0.47444 

0.78883 

0.77000 

0.69132 

Overload 

0.0606 

0.1321 

0.S256 

0.2112 

0.0429 

0.0854 

Density 

6 

4 

4 

4 

7 

7 

'emporal 

Demand 

Own 

Perfor¬ 

mance 

Effort 

Frus¬ 

tration 

Task 

Load 

Index 

Mean 

Suhscale 

Workload 

0.97880 

-0.85684 

0.97128 

0.97606 

0.98965 

0.98537 

0.0007 

0.0293 

0.0012 

0.0009 

0.0002 

0.0003 

6 

6 

6 

6 

6 

6 

0.99791 

-0.95961 

0.99122 

0.99966 

0.99913 

0.99880 

0.0021 

0.0404 

0.0088 

0.0003 

0.0009 

0.0012 

4 

4 

4 

4 

4 

4 

0.87797 

-0.86142 

0.90705 

0.85735 

0.86658 

0.87058 

0.1220 

0.1386 

0.0930 

0.1426 

0.1334 

0.1294 

4 

4 

4 

4 

4 

4 

0.98835 

-0.98659 

0.98942 

0.98602 

0.98785 

0.98818 

0.0116 

0.0134 

0.0106 

0.0140 

0.0122 

0.0118 

4 

4 

4 

4 

4 

4 

0.99032 

-0.92079 

0.97031 

0.99503 

0.98377 

0.99082 

0.0001 

0.0032 

0.0003 

0.0001 

0.0001 

0.0001 

7 

7 

7 

7 

7 

7 

0.95873 

-0.86974 

0.99360 

0.98361 

0.98850 

0.99183 

0.0006 

0.0110 

0.0001 

0.0001 

0.0001 

0.0001 

7 

7 

7 

7 

7 

7 

-0.92248 

0.96527 

0.98732 

0.98217 

0.98487 

0.0031 

0.0004 

0.0001 

0.0001 

0.0001 

7 

7 

7 

7 

7 

■0.92248 

-0.89319 

-0.92539 

-0.90591 

-0.89408 

0.0031* 

0.0068 

0.0028 

0.0050 

0.0066 

Root  MeanI 
Squared] 
Error 

Otnisslonj 

Error 


Commission] 

Error 


Exact 

Match 

Error 

Mental 

Demand 


Physical 

Demand 


Teirporal 

Demand 


Own 

Performance! 


Effort 


Frus  .atio; 


Task 

Load 

Index 

Mean 

Suhscale 

Workload 

Time-Based 
IB  I 


IB  I 
Standard 
Deviation 


0.96527 

0.0004 

7 

0.98732 

0.0001 

7 

0.98217 

0.0001 

7 

0.984S7 

0.0001 

7 

-0.72462 

0.0655 

7 

-0.34022 

0.4553 

7 


0.98867 

0.0001 

7 


-0.92539 

0.0028 

7 

-0.90591 

0.0050 

7 

-0.89408 

0.0066 

7 

0,74831 

0.0530 

7 

0.16367 

0.7258 

7 


0.99127 

0.0001 

7 

0.99128 

0.0001 

7 

-0.80040 

0.0306 

7 

-0.36623 

0.4191 

7 


0.99127  0.99128 

0.0001  0.0001 

7  7 

0.99351  0.99635 

0.0001  0.0001 

7  7 


0 . 9935lf^E|lfiu#'  0 . 99497 

0  0  •  0001 
7  7 


0.99635 

0,0001 

7 

-0.74573 

0.0543 

7 

-0.31674 

0.4888 

7 


0.99497;f^C:8M0$! 

o.oooii?^'^'; 

7 

-0.77552  -0.73372 
0.0405  0.0605 

7  7 

-0.36737  -0.37739 

0.4176  0.4040 

7  7 


IBI 
Coeff . 
Variance 


-0.04044 

0.9314 

7 


-0.12354 

0.7919 

7 


-0.04697 

0.9204 

7 


-0.01640 

0.9722 

7 


0.05312 

0.9099 

7 


-0.07813 

0.8678 

7 


5 


Temporal 

Demand 

Ovn 

Perfor¬ 

mance 

Effort 

Frus¬ 

tration 

Task 

Load 

Index 

Mean 

Subscale 

Workload 

RMS 

Succ.  Inter. 
Differences 

0.46745 

0.2902 

7 

-0.44891 

0.3123 

7 

0.40535 

0.3670 

7 

0.47188 

0.2850 

7 

0.41511 

0.3544 

7 

0.43384 

0.3308 

7 

RMS 
Cooff . 
Variance 

0.66611 

0.1023 

7 

-0.65626 

0.1079 

7 

0.61409 

0.1424 

7 

0.66066 

0.1062 

7 

0.62474 

0.1336 

7 

0.62372 

0.1345 

7 

Sum 

Succ.  Inter 
Differences 

0.65088 

0.1134 

7 

-0.56623 

0.1666 

7 

0.62373 

0.1344 

7 

0.67441 

0.0966 

7 

0.63944 

0.1220 

7 

0.64961 

0.1143 

7 

Decel.  / 
Fluctuations 

0.36243 

0.4243 

7 

-0.26289 

0.5690 

7 

0.31946 

0.4649 

7 

0.39129 

0.3854 

7 

0.33972 

0.4560 

7 

0.37405 

0.4085 

7 

Frequency- 

Based 

IBI 

-0.92008 

0.0033 

7 

0.85953 

0.0132 

7 

-0.54627 

0.2046 

7 

-0.86802 

0.0113 

7 

-0.88395 

0.0083 

7 

-0.85727 

0.0137 

7 

Respiration 

Band 

-0.71935 

0.0684 

7 

0.51401 

0.2379 

7 

-0.71038 

0.0736 

7 

-0.71333 

0.0719 

7 

-0.73193 

0.0615 

7 

-0.74759 

0.0534 

7 

Blood 

Pressure 

Band 

-0.08576 

0.8549 

7 

-0.22085 

0.6341 

7 

-0.45385 

0.3064 

7 

-0.08220 

0.8609 

7 

-0.14376 

0.7585 

7 

-0.16529 

0.7232 

7 

Model  1 
Cognitive 

0.64825 

0.0159 

7 

-0.82580 

0.0221 

7 

0.70068 

0.0795 

7 

0.77370 

0.0412 

7 

0.73831 

0.0581 

7 

0.75661 

0.0490 

7 

Model  1 
Psychomotor 

0.82679 

0.0218 

7 

-0.77377 

0.0412 

7 

0.94013 

0.0016 

7 

0.89538 

0.0064 

7 

0.90904 

0.0046 

7 

0.90539 

0.0050 

7 

Model  1 
Visual 

0.25075 

0.5876 

7 

-0.14879 

0.7502 

7 

0.41710 

0.3519 

7 

0.30939 

0.4995 

7 

0.36696 

0.4181 

7 

0.36184 

0.4251 

7 

Model  1 
Auditory 

0.36601 

0.4194 

7 

-0.38814 

0.3896 

7 

0.12552 

0.7886 

7 

0.24189 

0.6013 

7 

0.18924 

0.6845 

7 

0.21C3C 

0.6499 

7 

Model  1 
Kinesthetic 

0.84416 

0.0169 

7 

-0.79156 

0.0339 

7 

0.95008 

0.0010 

7 

0.90898 

0.0046 

7 

0.92096 

0.0032 

7 

0.91801 

0.0035 

7 

Model  1 
Mean 
Workload 

0.96578 

0.0004 

7 

-0.91112 

0.0043 

7 

0.93493 

0.0020 

7 

0.95005 

0.0010 

7 

0.94058 

0.0016 

7 

0.95120 

0.0010 

7 

Model  1 
Overload 
Conditions 

0.84835 

0.0156 

7 

-0.61583 

0.1409 

7 

0.73691 

0.0588 

7 

0.78230 

0.0376 

7 

0.78602 

0.0361 

7 

0.80514 

0.0289 

7 

Model  1 
Overload 
Density 

0.84726 

0.0161 

7 

-0.61354 

0.1428 

7 

0.73607 

0.0593 

7 

0.77870 

0.0391 

7 

0.78233 

0.0376 

7 

0.80190 

0.0301 

7 

F-6 


Model  2 


Model  2 


Modal  2 


Temporal 

Demand 

Own 

Perfor¬ 

mance 

Effort 

Frus¬ 

tration 

Task 

Load 

Index 

Mean 

Subscale 

Workload 

0.82731 

0.0216 

7 

-0.79544 

0.0324 

7 

0.66699 

0.1017 

7 

0.74671 

0.0538 

7 

0.70843 

0.0748 

7 

0.72753 

0.0639 

7 

0.87056 

0.0108 

7 

-0.78074 

0.0383 

7 

0.96526 

0.0004 

7 

0.92317 

0.0030 

7 

0.93839 

0.0018 

7 

0.93800 

0.0018 

7 

0.37201 

0.4112 

7 

-0.26323 

0.5684 

7 

0.49870 

0.2546 

7 

0.40689 

0.3650 

7 

0.45833 

0.3010 

7 

0.45797 

0.3014 

7 

0.36601 

0.4194 

7 

-0.38814 

0.3896 

7 

0.12552 

0.7886 

7 

0.24189 

0.6013 

7 

0.18924 

0.6845 

7 

0.21086 

0.6499 

7 

0.95870 

0.0007 

7 

-0.88023 

0.0089 

7 

0.90220 

0.0054 

7 

0.92403 

0.0029 

7 

0.91408 

0.0040 

7 

0.92789 

0.0026 

7 

0.74752 

0.0534 

7 

-0.55917 

0.1919 

7 

0.64811 

0.1154 

7 

0.71802 

0.0692 

7 

0.72219 

0.0668 

7 

0.73379 

0.0605 

7 

0.74752 

0.0534 

7 

-0.55917 

0.1919 

7 

0.64811 

0.1154 

7 

0.71802 

0.0692 

7 

0.72219 

0.0668 

7 

0.73379 

0.0605 

7 

F-: 


|SS\:'Stv 


Time-  IBl 

Based  Standard 


IBI  RMS  Succ.  RMS 

Coeff.  Inter.  Coeff. 


Root  Mean 
Squared 
Error 

Omission 

Error 


Commission 

Error 


Exact 

Match 

Error 

Mental 

Demand 


Physical 

Demand 


Temporal 

Demand 


Own 

Performance 


Effort 


Task 

Load 

Index 

Mean 

Subscale 

Workload 


-0.70355 

0.1188 

6 

-0.74479 

0.2552 

4 

-0.89126 

0.1087 

4 

-0.75446 

0.2455 

4 

-0.68553 

0.0891 

7 

-0.74225 

0.0561 

7 

-0.72462 

0.0655 

7 

0.74831 

0.0530 

7 

-0.80040 

0.0306 

7 


Frustration  -0.74573 
0.0543 


-0.77552 

0.0405 

7 

-0.73372 

0.0605 

7 


Tlme-Basedpl^P 
IBII  ' 


0.08364 

0.8748 

6 

-0.69428 

0.3057 

4 

-0.86405 

0.1360 

4 

-0.81643 

0.1816 

4 

-0.31001 

0.4986 

7 

-0.44021 

0.3229 

7 

-0.34022 

0.4553 

7 

0.16367 

0.7258 

7 

-0.36623 

0.4191 

7 

-0.31674 

0.4888 

7 

-0.36737 

0.4176 

7 

-0.37739 

0.4040 

7 

0.11577 

0.8048 

7 


IBI 

Standard 

Deviation 

IBI 

Coeff. 

Variance 


0. 11577 

0.8048tP  ## 

7 


-0.26558 

0.5649 

7 


0.92505 

0.0028 

1: 


Variance 

0.47264 

0.3438 

6 

0,90783 

0.0124 

6 

0.94123 

0.0051 

6 

-0.52429 

0.4757 

4 

0.19788 

0.8021 

4 

0.43161 

0.5684 

4 

-0.68243 

0.3176 

4 

-0.16771 

0.8323 

4 

0.14741 

0.8526 

4 

-0.66650 

0.3335 

4 

0.01127 

0.9887 

4 

0.26863 

0.7314 

4 

-0.02970 

0.9496 

7 

0.49572 

0.2579 

7 

0.66583 

0.1025 

7 

-0.14103 

0.7630 

7 

0.34723 

0.4454 

7 

0.54566 

0.2052 

7 

-0.04044 

0.9314 

7 

0.46745 

0.2902 

7 

0.66611 

0.1023 

7 

-0.12354 

0.7919 

7 

-0.44891 

0.3123 

7 

-0.65826 

0.1079 

7 

-0.04697 

0.9204 

7 

0.40535 

0.3670 

7 

0.61409 

0.1424 

7 

-0.01640 

0.9722 

7 

0.47188 

0.2850 

7 

0.66066 

0.1062 

7 

-0.05312 

0.9099 

7 

0.41511 

0.3544 

7 

0.62474 

0.1336 

7 

-0.07813 

0.8678 

7 

0.43384 

0.3308 

7 

0.62372 

0.1345 

7 

-0.26558 

0.5649 

7 

-0.32265 

0.4803 

7 

-0.60974 

0.1460 

7 

0.92505 

0.0028 

7 

0.60296 

0.1518 

7 

0.44055 

0.3225 

7 

0.73011 

0.0625 

0.68084 

0.0922 

RMS 

Succ.  Inter. 
Differences 


Tlme- 

Based 

IBI 

-0.32265 

0.4803 

1 


IBI 

Standard 

Deviation 

0.60296 

0.1518 

7 


IBI  RMS  Succ.  RMS 

Coeff.  Inter.  Coeff. 

Variance _ Diff.  Variance 


0.73011 

0.0625 

7 


0.93931 

0.0017 

7 


RMS 

Coeff. 

Variance 

-0.60974 

0.1460 

7 

0.44055 

0.3225 

7 

0.68084 

0.0922 

7 

0.93931 

0.0017 

7 

Sum 

Succ.  Intel 
Differences 

-0.46386 

0.2944 

7 

0.38142 

0.3985 

7 

0.56767 

0.1837 

7 

0.94113 

0.0016 

7 

0.93271 

0.0022 

7 

Decel.  / 
Fluctuations 

-0.03918 

0.9335 

7 

0.46850 

0.2890 

7 

0.49244 

0.2616 

7 

0.88712 

0.0077 

7 

0.73096 

0.0620 

Frequency- 

Based 

IBI 

0.60195 

0.1527 

7 

0.12708 

0.7860 

7 

-0.05658 

0.9041 

7 

0.09270 

0.8433 

7 

-0.00582 

0.9901 

7 

Respiration 

Band 

0.06819 

0.6845 

7 

0.77798 

0.0394 

7 

0.66891 

0.1004 

7 

0.85816 

0.0135 

7 

0.81672 

0.0250 

7 

Blood 

Pressure 

Band 

-0.34919 

0.4427 

7 

0.95751 

0.0007 

7 

0.93509 

0.0020 

7 

0.93955 

0.0017 

7 

0.91856 

0.0035 

7 

Model  1 
Cognitive 

-0.44167 

0.3211 

7 

-0.25188 

0.5858 

7 

-0.05576 

0.9055 

7 

0.42373 

0.3435 

7 

0.57319 

0.1786 

7 

i’lodel  1 
Psychomotor 

-0.75610 

0.0492 

7 

-0.42798 

0.3381 

7 

-0.13871 

0.7668 

7 

0.22308 

0.6306 

7 

0.41671 

0.3524 

7 

Model  1 
Visual 

-0.32958 

0.4704 

7 

-0.82675 

0.0218 

7 

-0.69878 

0.0807 

7 

-0.57909 

0.1731 

7 

-0.40435 

0.3683 

7 

Model  1 
Auditory 

0.02333 

0.9604 

7 

0.10661 

0.8200 

7 

0.12529 

0.7890 

7 

0.39521 

0.3802 

7 

0.41312 

0.3569 

7 

Model  1 
Kinesthetic 

-0.75992 

0.0474 

7 

-0.42699 

0.3393 

7 

-0.13533 

0.7724 

7 

0.23776 

0.6077 

7 

0.43317 

0.3316 

7 

Model  1 
Mean 
Workload 

-0.66460 

0,1034 

7 

-0.49191 

0.2622 

7 

-0.21809 

0.6385 

7 

0.28299 

0.5386 

7 

0.49506 

0.2586 

7 

Model  1 
Overload 
Conditions 

-0.41358 

0.3564 

7 

-0.37967 

0.4009 

7 

-0.17055 

0.7147 

7 

0.45840 

0.3009 

■7 

0.57406 

0.1777 

Model  1 
Overload 
Density 


0,57740 

0.1746 

7 


-0.42728 

0.3390 

7 


-0.38088 

0.3993 

7 


-0.16624 

0.7217 

7 


0.45611 

0.3037 

7 


Time- 

Based 

IBI 

XBX 

Standard 

Deviation 

XBX 
Coeff . 
variance 

BMS  Succ. 
Inter. 
Diff. 

RMS 

Coeff. 

Variance 

Model  2 
Cognitive 

-0.42362 

0.3436 

7 

-0.16732 

0.7199 

7 

0.02386 

0.9595 

7 

0.50079 

0.2523 

7 

0.63273 

0.1272 

7 

Model  2 
Psychomotor 

-0.77316 

0.0415 

7 

-0.45667 

0.3030 

7 

-0.15313 

0.7431 

7 

0.25719 

0.5777 

7 

0.45890 

0.3003 

7 

Model  2 
Visual 

-0.36361 

0.4227 

7 

-0.90015 

0.0057 

7 

-0.75208 

0.0512 

7 

-0.55085 

0.2000 

7 

-0.35053 

0.4408 

7 

Model  2 
Auditory 

0.02333 

0.9604 

7 

0.10661 

0.8200 

7 

0.12529 

0.7890 

7 

0.39521 

0.3802 

7 

0.41312 

0.3569 

7 

Model  2 
Mean 
Workload 

-0.63179 

0.1280 

7 

-0.49574 

0.2579 

7 

-0.22801 

0.6229 

7 

0.29469 

0.5212 

7 

0.50348 

0.2493 

7 

Model  2 
Overload 
Conditions 

-0.20688 

0.6563 

7 

-0.31574 

0.4903 

7 

-0.19553 

0.6744 

7 

0.42271 

0.3447 

7 

0.46074 

0.2981 

7 

Model  2 
Overload 
Density 

-0.20688 

0.6563 

7 

_ 

-0.31574 

0.4903 

7 

-0.19553 

0.6744 

7 

0.42271 

0.3447 

7 

0.46074 

0.2981 

7 

F-10 


F-11 


Sum  Succ. 

Decel.  / 

Frequency- 

Respir¬ 

Blood 

Inter 

Fluct . 

Based 

ation 

Pressure 

Diff. 

IBI 

Band 

Band 

RMS 

Succ.  Inter. 
Differences 

RMS 
Coeff . 
Variance 


0.94113 

0.0016 

7 

0.93271 

0.0022 

7 


0.88712 

0.0077 

7 

0.73096 

0.0620 

7 


0.09270 

0.8433 

7 

-0.00582 

0.9901 

7 


0.85816 

0.0135 

7 

0.81672 

0.0250 

7 


0.93955 

0.0017 

7 

0.91856 

0.0035 

7 


Sun 

Succ.  Inter 
Differences 

Decel.  / 
Fluctuations 


Frequency- 

Based 

IBI 


0.89839 
0.00  60 

0.15836 

0.7345 

7 


0.89839 

0.0060 

7 


-0.15836 

0.7345 

7 

-0.08987 

0.8480 

7 


-0.08987 

0.8480 

7 


0.62759 

0.1313 

7 

0.62742 

0.1315 

7 

0.55278 

0.1981 

7 


0.81610 

0.0252 

7 

0.78590 

0.0362 

7 

-0.08629 

0.8540 

7 


Respiration 

Band 


0.62759 

0.1313 


0.62742 

0.1315 

7 


0.55278 

0.1981 

7 


^ 


0.69432 

0.0835 

7 


Blood 

Pressure 

Band 

Model  1 
Cognitive 


Model  1 
Psychonotor 


Model  1 
Visual 


Model  1 
Auditory 


Model  1 
Kinesthetic 


Model  1 
Mean 
Workload 

Model  1 
Overload 
Conditions 

Model  1 
Overload 
Density 


0.81610 

0.0252 

7 

0.46624 

0.2916 

7 

0.48757 

0.2670 

7 

0.32277 

0.4801 

7 

0.25102 

0.5871 

7 

0.49900 

0.2543 

7 

0.46137 

0.2974 

7 

0.60328 

0.1515 

7 

0.59403 

0.1596 


0.78590 

0.0362 

7 

0.24896 

0.5903 

7 

0.21589 

0.6420 

7 

-0.46360 

0.2947 

7 

0.20258 

0.6631 

7 

0.22403 

0.6291 

7 

0.17709 

0.7041 

7 

0.45370 

0.3065 

7 

0.43645 

0.3276 


-0.08629 

0.8540 

7 

•0.89215 

0.0069 

7 

-0.67038 

0.0993 

7 

-0.74066 

0.0569 

7 

-0.07799 

0.8680 

7 

-0.25779 

0.5768 

7 

-0.77560 

0.0404 

7 

-0.75386 

0.0503 

7 

-0.79671 

0.0320 


0.69432 

0.0835 

7 

-0.77365 

0.0413 

7 

-0.18902 

0.6848 

7 

-0.73944 

0.0575 

7 

-0.17293 

0.7108 

7 

-0,66817 

0.1009 

7 

-0.21364 

0.6455 

7 

-0.58591 

0.1669 

7 

-0.73472 

0.0600 

7 

-0.29410 

0.5220 

7 

0.05270 

0.9107 

7 

-0.88717 

0.0077 

7 

-0.33652 

0.4605 

7 

-0.66215 

0.0914 

7 

-0.21317 

0.6463 

7 

-0.72213 

0.0669 

-0.29020 

0.5278 

Sum  Succ. 
Inter 
Diff. 

Decel .  / 
Fluct . 

Freqtuency- 

Based 

IBl 

Respir¬ 

ation 

Band 

Blood 

Pressure 

Band 

Model  2 
Cognitive 

0.52100 

0.2305 

7 

0.31679 

0.4886 

7 

-0.67473 

0.0964 

7 

-0.67235 

0.0980 

7 

-0.10537 

0.8221 

7 

Model  2 
Psychonudtor 

0.51635 

0.2354 

7 

0.23638 

0.6098 

7 

-0.77325 

0.0414 

7 

-0.71271 

0.0723 

7 

-0.26517 

0.5655 

7 

Model  2 
Visual 

-0.29939 

0.5142 

7 

-0.46927 

0.2881 

7 

-0.15148 

0.7458 

7 

-0.72979 

0.0626 

7 

-0.81040 

0.0271 

7 

Model  2 
Auditory 

0.25102 

0.5871 

7 

0.20258 

0.6631 

7 

-0.25779 

0.5768 

7 

-0.29410 

0.5220 

7 

0.05270 

0.9107 

7 

Model  2 
Mean 
Workload 

0.45169 

0.3090 

7 

0.17535 

0.7069 

7 

-0.75949 

0.0476 

7 

-0.89506 

0.0065 

7 

-0.37091 

0.4127 

7 

Model  2 
Overload 
Conditions 

0.62528 

0.1332 

7 

0.58416 

0.1685 

7 

-0.82035 

0.0238 

7 

-0.61407 

0.1424 

7 

-0.09440 

0.8405 

7 

Model  2 
Overload 
Density 

0.62528 

0.1332 

7 

0.58416 

0.1685 

7 

-0.82035 

0.0238 

7 

-0.61407 

0.1424 

7 

-0.09440 

0.8405 

7 

F-13 


^  '  '  ’  < 

Model  1 
Cognitive 

Model  1 
Psychomotor 

Model  1 
Visual 

Model  1 
Auditory 

Root  Mean 

0.76029 

0.85620 

-0.44989 

0.44989 

Squared 

0.0793 

0.0295 

0.3707 

0.3707 

Error 

6 

6 

6 

6 

omission 

0.94972 

0.98337 

0.67257 

Error 

0.0503 

0.0166 

0.3274 

4 

4 

4 

4 

Commission 

0.93752 

0.90784 

0.89232 

Error 

0.0625 

0.0922 

0.1077 

4 

4 

4 

4 

Exact 

0.99216 

0.99993 

0.80206 

Match 

0.0078 

0.0001 

0.1979 

Error 

4 

4 

4 

4 

Mental 

0.80940 

0.85851 

0.25576 

0.30742 

Demand 

0.0274 

0.0134 

0.5799 

0.5024 

7 

7 

7 

7 

Physical 

0.68945 

0.94847 

0.46989 

0.10142 

Demand 

0.0866 

0.0011 

0.2874 

0.8287 

7 

7 

7 

7 

Tanporal 

0.84825 

0.82679 

0.25075 

0.36601 

Demand 

0.0159 

0.0218 

0.5876 

0.4194 

7 

7 

7 

7 

Own 

-0.82580 

-0.77377 

-0.14879 

-0.38814 

Performance 

0.0221 

0.0412 

0.7502 

0.3896 

7 

7 

7 

7 

Effort 

0.70068 

0.94013 

0.41710 

0.12552 

0.0795 

0.0016 

0.3519 

0.7886 

7 

7 

7 

7 

Frustration 

0.77370 

0.89538 

0.30939 

0.24189 

0.0412 

0.0064 

0.4995 

0.6013 

7 

7 

7 

7 

Task 

0.73831 

0.90904 

0.36696 

0.18924 

Load 

0.0581 

0.0046 

0.4181 

0.6845 

Index 

7 

7 

7 

7 

Mean 

0.75661 

0.90539 

0.36184 

0.21086 

Subscale 

0.0490 

0.0050 

0.4251 

0.6499 

Workload 

7 

7 

7 

7 

Time-Based 

-0.44167 

-0.75610 

-0.32958 

0.02333 

IBI 

0.3211 

0.0492 

0.4704 

0.9604 

7 

7 

7 

7 

IB  I 

-0.25188 

-0.42798 

-0.82675 

0.10661 

Standard 

0.5858 

0.3381 

0.0218 

0.8200 

Deviation 

7 

7 

7 

7 

IBI 

-0.05576 

-0.13871 

-0.69878 

0.12529 

Coeff . 

0.9055 

0.7668 

0.0807 

0.7890 

Variance 

L  -  —  -  — 

7 

7 

7 

7 

F-14 


Model  1  Model  1 

Cognitive  Psychomotor 


Model  1 
Vlsuel 


Model  1 
Auditory 


Succ .  Inte 


Frequency- 


Re  spiratio 


Model 


0.42373 

0.3435 

1  7 

0.22308 

0.6306 

7 

-0.57909 

0.1731 

7 

0.39521 

0.3802 

7 

0.57319 

0.1786 

7 

0.41671 

0.3524 

7 

-0.40435 

0.3683 

7 

0.41312 

0.3569 

7 

0.46624 
r  0.2916 
}  7 

0.48757 

0.2670 

7 

-0.32277 

0.4801 

7 

0.25102 

0.5871 

7 

0.24896 
i  0.5903 

7 

0.21589 

0.6420 

7 

-0.46360 

0.2947 

7 

0.20258 

0.6631 

7 

-0.89215 

-0.67038 

-0.74066 

-0.07799 

1  0.0069 

7 

0.0993 

7 

0.0569 

7 

0.8680 

7 

1  -0.77365 

0.0413 
7 

-0.73944 

0.0575 

7 

-0.66817 

0.1009 

7 

-0.58591 

0.1669 

7 

-0.18902 

0.6848 

7 

-0.17293 

0.7108 

7 

-0.21364 

0.6455 

7 

-0.73472 

0.0600 

7 

0.44994 

0.3111 

7 

-0.06653 

0.8873 

m 

/ 

0.78239 

0.0376 

7 

0.449941 

0.3111i 

71 

0.62483 

0.1336 

7 

-0.19682 

0.6723 

7 

-0.06653 

0.8873 

7 

-0.57085 

0.1807 

7 

0.78239 

0.0376 

7 

-0.19682 

0.6723 

7 

-0.570851 

0.1807* 

l\ 

0.47905 

0.2767 

7 

0.99945 

0.0001 

7 

0.60939 

0.1463 

7 

-0.16432 

0.7248 

7 

0.87052 

0.0108 

7 

0.81772 

0.0246 

7 

0.37369 

0.4090 

7 

0.37571 

0.4062 

7 

0.79786 

0.0315 

7 

0.52458 

0.2267 

7 

0.07830 

0.8675 

7 

0.51096 

0.2412 

7 

0.80294 

0.0297 

0.51920 

0.2324 

0.07954 

0.8654 

0.51904 

0.2326 

Density 


Model  1 
Cognitive 

Model  1 
Psychomotor 

Model  1 
visual 

Model  1 
Auditory 

Model  2 

0.99330 

0.39409 

-0.16231 

0.82138 

Cognitive 

0.0001 

0.3817 

0.7281 

0.0235 

7 

7 

7 

7 

Model  2 

0.51248 

0.99131 

0.60447 

-0.12464 

Psychomotor 

0.2396 

0.0001 

0.1505 

0.7900 

7 

7 

7 

7 

Model  2 

0.11987 

0.65141 

0.97896 

-0.39130 

Visual 

0.7980 

0.1130 

0.0001 

0.3854 

7 

7 

7 

7 

Model  2 

0.78239 

-0.19682 

-0.57085 

1,00000 

Auditory 

0.0376 

0.6723 

0.1807 

0.0001 

7 

7 

7 

7 

Model  2 

0.90973 

0.75014 

0.31739 

0.46261 

Mean 

0.0045 

0.0521 

0.4879 

0.2959 

Workload 

7 

7 

7 

7 

Model  2 

0.63542 

0.51426 

0.05418 

0.35355 

Overload 

0.1251 

0.2377 

0.9082 

0.4366 

Conditions 

7 

7 

7 

7 

Model  2 

0.63542 

0.51426 

0.05418 

0.35355 

Overload 

0.1251 

0.2377 

0.9082 

0.4366 

Density 

7 

7 

7 

7 

F-16 


Model  1 
Kinesthetic 

Model  1 
Mean 
Workload 

Model  1 
Overload 
Condtions 

Model  1 
Overload 
Densitv 

Root  Mean 

0.87234 

0.90495 

0.85581 

0.85102 

Squared 

0.0234 

0.C131 

0.0297 

0.0316 

Error 

6 

6 

6 

6 

Omission 

0.98337 

0.93697 

0.97689 

0.97356 

Error 

0.0166 

0.0630 

0.0231 

0.0264 

4 

4 

4 

4 

Commission 

0.90784 

0.94279 

0.78305 

0.80227 

Error 

0.0922 

0.0572 

0.2169 

0.1977 

4 

4 

4 

4 

Exact 

0.99993 

0.98662 

0.92365 

0.92365 

Match 

0.0001 

0.0134 

0.0763 

0.0764 

Error 

4 

4 

4 

4 

Mental 

0.87410 

0.95146 

0.81812 

0.81263 

Demand 

0.0101 

0.0010 

0.0245 

0.0263 

7 

7 

7 

7 

Physical 

0.95766 

0.93965 

0.74946 

0.74570 

Demand 

0.0007 

0.0017 

0.0524 

0.0543 

7 

7 

7 

7 

Teiiporal 

0.84416 

0.96578 

0.84835 

0.84726 

Demand 

0.0169 

0.0004 

0.0158 

0.0161 

7 

7 

7 

7 

Own 

-0.79156 

-0.91112 

-0.61583 

-0.61354 

Performance 

0.0339 

0.0043 

0.1409 

0.1428 

7 

7 

7 

7 

Effort 

0.95008 

0.93493 

0.73691 

0.73607 

0.0010 

0.0020 

0.0588 

0.0593 

7 

7 

7 

7 

Frustration 

0.90898 

0.95005 

0.78230 

0.77870 

0.0046 

0.0010 

0.0376 

0.0391 

7 

7 

7 

7 

Task 

0.92096 

0.94058 

0.78602 

0.78233 

Load 

0.0032 

0.0016 

0.0361 

0.0376 

Index 

7 

7 

7 

7 

Mean 

0.91801 

0.95120 

0.80514 

0.80190 

Subscale 

0.0035 

0.0010 

0.0289 

0.0301 

Workload 

7 

7 

7 

7 

Time-Based 

-0.75992 

-0.66460 

-0.41358 

-0.42728 

IBI 

0.0474 

0.1034 

0.3564 

0.3390 

7 

7 

7 

7 

IBI 

-0.42699 

-0.49191 

-0.37967 

-0.38088 

Standard 

0.3393 

0.2622 

0.4009 

0.3993 

Deviation 

7 

7 

7 

7 

IBI 

-0.13533 

-0.21809 

-0.17055 

-0.16624 

Coeff . 

0.7724 

0.6385 

0.7147 

0.7217 

Variance 

7 

7 

7 

7 

r-17 


Model  1 
Kinesthetic 


Model  1 
Mean 
Workload 


Model  1 
Overload 
Condtions 


Model  1 
Overload 
Density 


RMS 

0.23776 

0.28299 

0.45840 

0.45611 

Succ.  Inter. 

0.6077 

0.5386 

0.3009 

0.3037 

Differences 

7 

7 

7 

7 

RMS 

0.43317 

0.49506 

0.57406 

0.57740 

Coeff . 

0.3316 

0.2586 

0.1777 

0.1746 

Variance 

7 

7 

7 

7 

Suit 

0.49900 

0.46137 

0.60328 

0.59403 

Succ.  Inter 

0.2543 

0.2974 

0.1515 

0.1596 

Differences 

7 

7 

7 

7 

Decel.  / 

0.22403 

0.17709 

0.45370 

0.43645 

Fluctuations 

0.6291 

0.7041 

0.3065 

0.3276 

7 

7 

7 

7 

Frequency- 

-0.25779 

-0.77560 

-0.75386 

-0.79671 

Based 

0.5768 

0.0404 

0.0503 

0.0320 

IBI 

7 

7 

7 

7 

Respiration 

-0.29410 

-0.88717 

-0.68215 

-0.72213 

Band 

0.5220 

0.0077 

0.0914 

0.0669 

7 

7 

7 

7 

Blood 

Pressure 

Band 


0.05270 

0.9107 

7 


-0.33652 

0.4605 

7 


-0.21317 

0.6463 

7 


-0.29020 

0.5278 

7 


Model  1 

0.47905 

0.87052 

0.79786 

0.80294 

Cognitive 

0.2767 

0.0108 

0.0315 

0.0297 

7 

7 

7 

7 

Model  1 

0.99945 

0.81772 

0.52458 

0.51920 

Psychomotor 

0.0001 

0.0246 

0.2267 

0.2324 

7 

7 

7 

7 

Model  1 

0.60939 

0.37369 

0.07830 

0.07954 

Visual 

0.1463 

0.4090 

0.8675 

0.8654 

7 

7 

7 

7 

Model  1 

-0.16432 

0.37571 

0.51096 

0.51904 

Auditory 

0.7248 

0.4062 

0.2412 

0.2326 

7 

7 

7 

7 

Model  1 
Kinesthetic 


0.63536 

0.0193 

7 


0.54500 

0.2058 

7 


0.53986 

0.2110 

7 


Model  1 
Mean 
Workload 


0.83536 

0.0193 

7 


0.78219 

0.0377 

7 


0.78392 

0.0370 

7 


Model  1 
Overload 
Conditions 


0.54500 

1  0.99906 

0.2058 

O.OOOl 

7 

1  7 

Model  1 
Overload 
Density 


0.53986 

0.2110 

7 


0.78392 

0.0370 

7 


Model  1 
Kinesthetic 

Model  1 
Mean 
Workload 

Model  1 
Overload 
Condtions 

Model  1 
Overload 
Density 

Model  2 
Cognitive 

0.42418 

0.3429 

7 

0.82702 

0.0217 

7 

0.81113 

0.0268 

7 

0.81686 

0.0249 

7 

Model  2 
Psychomotor 

0.99314 

0.0001 

7 

0.85499 

0.0142 

7 

0.61723 

0.1398 

7 

0.61466 

0.1419 

7 

Model  2 
Visual 

0.64218 

0.1199 

7 

0.51222 

0.2399 

7 

0.21475 

0.6438 

7 

0.21815 

0.6384 

7 

Model  2 
Auditory 

-0.16432 

0.7248 

7 

0.37571 

0.4062 

7 

0.51096 

0.2412 

7 

0.51904 

0.2326 

7 

Model  2 
Mean 
Workload 

0.77030 

0.0427 

7 

0.99152 

0.0001 

7 

0.83156 

0.0204 

7 

0.83594 

0.0191 

7 

Model  2 
Overload 
Conditions 

0.52931 

0.2218 

7 

0.65886 

0.1075 

7 

0.87727 

0.0095 

7 

0.85562 

0.0141 

7 

Model  2 
Overload 
Density 

0.52931 

0.2218 

7 

0.65886 

0.1075 

7 

0.87727 

0.0095 

7 

0.85562 

0.0141 

7 

F-19 


Model  2 
Cognitive 

Model  2 
Psychomotor 

Model  2 
Visual 

Model  2 
Auditory 

Root  Mean 

0.77810 

0.91949 

-0.44989 

0.44989 

Squared 

0.0684 

0.0095 

0.3707 

0.3707 

Error 

6 

6 

6 

6 

Omission 

0.98520 

0.98520 

0.67257 

Error 

0.0148 

0.0148 

0.3274 

4 

4 

4 

4 

Commission 

0.92383 

0.92383 

0.89232 

Error 

0.0762 

0.0762 

0.1077 

4 

4 

4 

4 

Exact 

0.99548 

0.99548 

0.80206 

Match 

0.0045 

0.0045 

0.1979 

Error 

4 

4 

4 

4 

Mental 

0.78626 

0.89018 

0.36306 

0,30742 

Demand 

0.0360 

0.0072 

0.4234 

0.5024 

7 

7 

7 

7 

Physical 

0.64913 

0.97165 

0.55187 

0.10142 

Demand 

0.1147 

0.0003 

0.1990 

0.8287 

7 

7 

7 

7 

Tenporal 

0.82731 

0.87056 

0.37201 

0.36601 

Demand 

0.0216 

0.0108 

0.4112 

0.4194 

7 

7 

7 

7 

Own 

-0.79544 

-0.78074 

-0.26323 

-0.38814 

Performance 

0.0324 

0.0383 

0.5684 

0.3896 

7 

7 

7 

7 

Effort 

0.66699 

0.96526 

0.49870 

0.12552 

0.1017 

0.0004 

0.2546 

0.7886 

7 

7 

7 

7 

Frustration 

0.74671 

0.92317 

0.40689 

0.24189 

0.0538 

0.0030 

0.3650 

0.6013 

7 

7 

7 

7 

Task 

0.70843 

0.93839 

0.45833 

0.18924 

Load 

0,0748 

0.0018 

0.3010 

0.6845 

Index 

7 

7 

7 

7 

Mean 

0.72753 

0.93800 

0.45797 

0.21086 

Subscale 

0,0639 

0.0018 

0.3014 

0.6499 

Workload 

7 

7 

7 

7 

Time-Based 

-0,42362 

-0.77316 

-0.36361 

0.02333 

IBI 

0.3436 

0.0415 

0.4227 

0.9604 

7 

7 

7 

7 

IBI 

-0.16732 

-0.45667 

-0.90015 

0.10661 

Standard 

0.7199 

0.3030 

0.0057 

0.8200 

Deviation 

7 

7 

7 

7 

IBI 

0.02386 

-0.15313 

-0.75208 

0.12529 

Coeff . 

0.9595 

0.7431 

0.0512 

0.7890 

Variance 

7 

7 

7 

7 

F-20 


Model  2 
Cognitive 

Model  2 
Psychomotor 

Model  2 
Visual 

Model  2 
Auditory 

RMS 

Succ.  Inter. 
Differences 

0.50079 

0.2523 

7 

0.25719 

0.5777 

7 

-0.55085 

0.2000 

7 

0.39521 

0.3802 

7 

RMS 
Coeff . 
Variance 

0.63273 

0.1272 

7 

0.45890 

0.3003 

7 

-0.35053 

0.4408 

7 

0.41312 

0.3569 

7 

Sum 

Succ.  Inter 
Differences 

0.52100 

0.2305 

7 

0.51635 

0.2354 

7 

-0.29939 

0.5142 

7 

0.25102 

0.5871 

7 

Decel.  / 
Fluctuations 

0.31679 

0.4888 

7 

0.23638 

0.6098 

7 

-0.46927 

0.2881 

7 

0.20258 

0.6631 

7 

Frequency- 
Based 
IB  I 

-0.67473 

0.0964 

7 

-0.77325 

0.0414 

7 

-0.15148 

0.7458 

7 

-0.25779 

0.5768 

7 

Respiration 

Band 

-0.67235 

0.0980 

7 

-0.71271 

0.0723 

7 

-0.72979 

0.0626 

7 

-0.29410 

0.5220 

7 

Blood 

Pressure 

Band 

-0.10537 

0.8221 

7 

-0.26517 

0.5655 

7 

-0.81040 

0.0271 

7 

0.05270 

0.9107 

7 

Model  1 
Cognitive 

0.99330 

0.0001 

7 

0.51248 

0.2396 

7 

0.11987 

0.7980 

7 

0.78239 

0.0376 

7 

Model  1 
Psychomotor 

0.39409 

0.3817 

7 

0.99131 

0.0001 

7 

0.65141 

0.1130 

7 

-0.19682 

0.6723 

7 

Model  1 
Visual 

-0.16231 

0.7281 

7 

0.60447 

0.1505 

7 

0.97896 

0.0001 

7 

-0.57085 

0.1807 

7 

Model  1 
Auditory 

0.82138 

0.0235 

7 

-0.12464 

0.7900 

7 

-0.39130 

0.3854 

7 

1.00000 

0.0001 

7 

Model  1 
Kinesthetic 

0.42418 

0.3429 

7 

0.99314 

0.0001 

7 

0.64218 

0.1199 

7 

-0.16432 

0.7248 

7 

Model  1 
Mean 
Workload 

0.82702 

0.0217 

7 

0.85499 

0.0142 

7 

0.51222 

0.2399 

7 

0.37571 

0.4062 

7 

Model  1 
Overload 
Conditions 

0,81113 

0.0268 

7 

0.61723 

0.1398 

7 

0.21475 

0.6438 

7 

0.51096 

0.2412 

7 

Model  1 
Overload 
Density 

0.81686 

0.0249 

0.61466 

0.1419 

0.21815 

0.6384 

0.51904 

0.2326 

r-21 


Model  2 
Cognitive 


Model  2 
Psychomotor 


Model  2 
Visual 


Model  2 
Auditory 


Model 

Cognitive  ,  , 


7 

0.46355 

0.2948 

7 


0.02221 

0.9623 

7 


0.82138 

0.0235 

7 


Model  2 
Fsychomotorl 


0.46355 

0.2948 

7 


0.64653 

0.1166 

7 


-0.12464 

0.7900 

7 


Model  2 
Visual 


0.02221 

0.9623 

7 


0.64653 

0.1166t 

7 


-0.39130 

0.3854 

7 


Model  2 
Auditory 


Model  2 
Mean 
Workload 

Model  2 
Overload 
Conditions 

Model  2 
Overload 
Density 


0.82138 

0.0235 

7 

0.87438 

0.0100 

7 

0.63969 

0.1218 

7 

0.63969 

0.1218 

7 


0.12464 

-0.39130 

0.7900 

0.3854 

7 

7 

0.80236 

0.47071 

0.0299 

0.2864 

7 

7 

0.56353 

0.14859 

0.1877 

0.7505 

7 

7 

0.56353 

0.14859 

0.1877 

0.7505 

7 

7 

0.46261 

0.2959 

7 

0.35355 

0.4366 

7 

0.35355 

0.4366 

7 


r-22 


Root  Mean 
Squared 
Error 

Omission 

Error 


Commission 

Error 


Exact 

Match 

Error 

Mental 

Demand 


Physical 

Demand 


Tenporal 

Demand 


Own 

Performance 


Effort 


Frustration 


Task 

Load 

Index 

Mean 

Subscale 

Workload 

Time-Based 

IBI 


IB  I 
Standard 
Deviation 

IBI 
Coeff . 
Variance 


Model  2 
Mean 
Workload 

Model  2 
Overload 
Conditions 

Model  2 
Overload 
Densitv 

0.88511 

0.0190 

6 

0.79158 

0.0606 

6 

0.79158 

0.0606 

6 

0.93865 

0.0614 

4 

0.86791 

0.1321 

4 

0.86791 

0.1321 

4 

0.96050 

0.0395 

4 

0.47444 

0.5256 

4 

0.47444 

0.5256 

4 

0.98604 

0.0140 

4 

0.78883 

0.2112 

4 

0.78883 

0.2112 

4 

0.93182 

0.0022 

7 

0.77000 

0.0429 

7 

0.7700C 

0.0429 

7 

0.90516 

0.0051 

7 

0.69132 

0.0854 

7 

0.69132 

0.0854 

7 

0.95870 

0.0007 

7 

0.74752 

0.0534 

7 

0.74752 

0.0534 

7 

-0.88023 

0.0089 

7 

-0.55917 

0.1919 

7 

-0.55917 

0.1919 

7 

0.90220 

0.0054 

7 

0.64811 

0.1154 

7 

0.64811 

0.1154 

7 

0.92403 

0.0029 

7 

0.71802 

0.0692 

7 

0.71802 

0.0692 

7 

0.91408 

0.0040 

7 

0.72219 

0.0668 

7 

0.72219 

0.0668 

7 

0.92789 

0.0026 

7 

0.73379 

0.0605 

7 

0.73379 

0.0605 

7 

-0.63179 

0.1280 

7 

-0.20688 

0.6563 

7 

-0.20688 

0.6563 

7 

-0.49574 

0.2579 

7 

-0.31574 

0.4903 

7 

-0.31574 

0.4903 

7 

-0.22801 

0.6229 

7 

-0.19553 

0.6744 

7 

-0.19553 

0.6744 

7 

F-23 


Model  2 
Mean 
Workload 


Model  2 
Overload 
Conditions 


Model  2 
Overload 
Density 


RMS 

0.29469 

0.42271 

0.42271 

Succ.  Inter. 

0.5212 

0.3447 

0.3447 

Differences 

7 

7 

7 

RMS 

0.50348 

0.46074 

0.46074 

Coeff . 

0.2493 

0.2981 

0.2981 

Variance 

7 

7 

7 

Sum 

0.45169 

0.62528 

0.62528 

Succ.  Inter 

0.3090 

0.1332 

0.1332 

Differences 

7 

7 

7 

Decel.  / 

0.17535 

0.58416 

0.58416 

Fluctuations 

0.7069 

0.1685 

0.1685 

7 

7 

7 

Frequency- 

-0.75949 

-0.82035 

-0.82035 

Based 

0.0476 

0.0238 

0.0238 

IBI 

7 

7 

7 

Respiration 

-0.89506 

-0.61407 

-0.61407 

Band 

0.0065 

0.1424 

0.1424 

7 

7 

7 

Blood 

-0.37091 

-0.09440 

-0.09440 

Pressure 

0.4127 

0.8405 

0.8405 

Band 

7 

7 

7 

Model  1 

0.90973 

0.63542 

0.63542 

Cognitive 

0.0045 

0.1251 

0.1251 

7 

7 

7 

Model  1 

0.75014 

0.51426 

0.51426 

Psychomotox 

0.0521 

0.2377 

0.2377 

7 

7 

7 

Model  1 

0.31739 

0.05418 

0.05418 

Visual 

0.4879 

0.9082 

0.9082 

7 

7 

7 

Model  1 

0.46261 

0.35355 

0.35355 

Auditory 

0.2959 

0.4366 

0.4366 

7 

7 

7 

Model  1 

0.77030 

0.52931 

0.52931 

Kinesthetic 

0.0427 

0.2218 

0.2218 

7 

7 

7 

Model  1 

0.99152 

0.65886 

0.65886 

Mean 

0.0001 

0.1075 

0.1075 

Workload 

7 

7 

7 

Model  1 

0.83156 

0.87727 

0.87727 

Overload 

0.0204 

0.0095 

0.0095 

Conditions 

7 

7 

7 

Model  1 

0.83594 

0.85562 

0.85562 

Overload 

0.0191 

0.0141 

0.0141 

Density 

7 

7 

F-24 


Model  2 
Mean 
Workload 


Model  2 
Overload 
Conditions 


Model  2 
OverloadI 
Density 


Model  2 
Cognitive] 

Model  2 
Psychomotorl 


Model  2 
Visual 


Model  2 
Auditory 


Model  2 
Mean 
Workload! 

Model  2 
Overload] 
Conditions 

Model  2 
OverloadI 
Density] 


0.87438 

0.0100 

7 

0.80236 

0.0299 

7 

0.47071 

0.2864 

7 

0.46261 

0.2959 

7 


0.67241 

0.0979 

7 

0.67241 

0.0979 

7 


0.63969 

0.1218 

7 

0.56353 

0.1877 

7 

0.14859 

0.7505 

7 

0.35355 

0.4366 

7 

0.67241 

0.0979 

7 


0.63969 

0.1218 

7 

0.56353 

0.1877 

7 

0.14859 

0.7505 

7 

0.35355 

0.4366 

7 

0.67241 

0.0979 

7 

1.00000 

0.0 

7 


F-25 


